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Overview 


Many students who enter community college are deemed underprepared for college-level courses and 
are referred to developmental (remedial) education courses to build their math, reading, or writing 
skills. These students often struggle in developmental courses and in college more broadly. To help 
them, the City University of New York (CUNY) developed CUNY Start. CUNY Start targets incom- 
ing students who are assessed as needing remediation in math, reading, and writing. The program 
delays college matriculation (enrollment in a degree program) for one semester and provides intensive 
instruction in math, reading, and writing during that semester with a prescribed instructional approach. 
It also provides advising, tutoring, and a weekly seminar that teaches students skills they need to suc- 
ceed in college. Students pay only $75 for the program and do not use financial aid. 


CUNY Start’s underlying theory of change posits that students with substantial developmental course 
requirements are best served through an intensive model, designed to build academic preparedness 
and college skills before matriculation. The program’s designers hypothesize that compared with stu- 
dents in standard college courses (including standard developmental education courses), a higher pro- 
portion of CUNY Start students will complete developmental education and that they will do so more 
quickly. Because CUNY Start students spend a semester building their basic skills before matriculat- 
ing, they are expected to earn fewer college credits in the short term. Over the longer term, the expec- 
tation is that CUNY Start students will have higher retention rates (that is, more of them will stay in 
college), higher college-level credit accumulation, and higher graduation rates. 


MDRC, CUNY, and the Community College Research Center at Teachers College, Columbia Uni- 
versity, are partnering to evaluate CUNY Start using a random assignment research design, supported 
by a grant from the federal Institute of Education Sciences. Eligible students at four CUNY commu- 
nity colleges were assigned at random to the program group, whose members could participate in 
CUNY Start, or to the control group, whose members could receive the colleges’ standard courses and 
services, including standard developmental education courses. Findings in this report include: 


e CUNY Start was implemented as it was designed, and the contrast between the program and the 
colleges’ standard courses and services was substantial. 


e During the first semester in the study, program group students made substantially more progress 
through developmental education than control group students; effects were especially large in 
math. In contrast, during that same semester, control group students earned more college credits 
than program group students, as predicted by CUNY Start’s designers. 


e During the second semester, program group students enrolled at CUNY colleges (that is, partici- 
pated in CUNY Start or enrolled in any non-CUNY Start courses as matriculated students) at a 
higher rate than control group students. 


Subsequent follow-up data will be analyzed to assess sample members’ persistence in college, college 
credit accumulation, and graduation rates. If CUNY Start’s short-term trade-off results in the hypoth- 
esized longer-term gains, the program will serve as an important model for serving students with sub- 
stantial developmental course requirements. 
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Preface 


Community colleges play a vital role in postsecondary education and workforce development, 
enrolling more than one in every three undergraduates in the United States. Unfortunately, many 
entering community college students are assessed as needing remediation in math, reading, or 
writing and are placed into noncredit developmental (remedial) education courses. Graduation 
rates for students who place into developmental education are discouragingly low. Yet seriously 
tackling the issue of pedagogical reform in remedial education classrooms (and higher education 
classrooms more broadly) may require rethinking the hiring, promotion, and professional devel- 
opment of instructors — no small undertaking. 


In 2009, the City University of New York (CUNY) developed and implemented CUNY 
Start, an innovative prematriculation program (one that precedes students’ entry into degree pro- 
grams) that redefines students’ experiences with developmental education. CUNY Start is time- 
intensive, changes the pedagogy used in remedial classes, modifies the content of developmental 
education, and gives students additional academic and nonacademic support. By focusing on de- 
velopmental education alone, it seeks to eliminate or dramatically reduce students’ developmental 
education requirements in one semester, helping clear their path to a degree. 


This report presents important early findings from a partnership anong MDRC; the Com- 
munity College Research Center at Teachers College, Columbia University; and CUNY to eval- 
uate the effectiveness of CUNY Start. The program served students who placed into developmen- 
tal education in one or more areas, and half of the study sample placed into remediation in math, 
reading, and writing. CUNY Start’s effect on becoming “college-ready” is among the largest any 
of the partner organizations has found in evaluations of developmental education reforms. How- 
ever, in line with CUNY Start’s theory of change, students offered the program do not take col- 
lege-level courses, and therefore fall behind in college credits earned before they matriculate. 
Additional data collection that is planned for the longer term will show whether this short-term 
trade-off is worthwhile. 


CUNY Start shows that it is possible to dramatically change the student experience inside 
the developmental education classroom — particularly in remedial math. Moreover, with the right 
combination of reform in the intensity, pedagogy, and content of instruction, many more students 
can become college-ready within a single semester. As the evaluation tracks these students into 
the future, much will be learned about this promising model. 


Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Many students who enter community college are deemed underprepared for college-level courses 
and are referred to developmental (remedial) education courses to build their math, reading, or 
writing skills. These students often struggle in the developmental courses and in college more 
broadly.! 


Hoping to boost the success rates of its least prepared incoming students, the City Uni- 
versity of New York (CUNY) developed CUNY Start, which is offered at eight CUNY colleges. 
CUNY Start’s full-time program was designed for and targets incoming students who are as- 
sessed as needing remediation in math, reading, and writing. It is also open to students who are 
assessed as needing remediation in math and either reading or writing. (CUNY Start is also of- 
fered part time, as discussed below.) The program’s short-term goal is to substantially reduce or 
eliminate students’ developmental education requirements after one semester, while preparing 
them for college-level courses. Its long-term goal is to improve academic outcomes, including 
graduation rates. CUNY Start delays college matriculation (that is, when students first enroll in 
non-CUNY Start courses in degree programs) for one semester and provides intensive instruction 
in math, reading, and writing during that semester with a prescribed instructional approach. It also 
provides advising, tutoring, and a weekly seminar that teaches students skills they need to succeed 
in college. Students pay only $75 for the program and cannot use financial aid. 


Compared with many developmental education reforms, CUNY Start is uncommonly 
comprehensive. The program’s focus on students assessed as needing remediation in math, read- 
ing, and writing sets it apart from other reforms that focus on students with remedial requirements 
in only one or two subject areas, or that focus on students on the cusp of being deemed “college- 
ready.”* Additionally, unlike most other reforms, CUNY Start aims to provide all the develop- 
mental education students need in one semester, before they matriculate. Finally, the cost is very 
low. 


‘Thomas Bailey, Dong Wook Jeong, and Sung-Woo Cho, “Referral, Enrollment, and Completion in Devel- 
opmental Education Sequences in Community Colleges” (Economics of Education Review 29, 2: 255-270, 
2010). 

*Michelle Hodara and Shanna Smith Jaggars, “An Examination of the Impact of Accelerating Community 
College Students’ Progression Through Developmental Education” (Journal of Higher Education 85, 2: 246- 
276, 2014); Nikki Edgecombe, Shanna Smith Jaggars, Di Xu, and Melissa Barragan, “Accelerating the Inte- 
grated Instruction of Developmental Reading and Writing at Chabot College” (New York: Community College 
Research Center, Teachers College, Columbia University, 2014); and Colleen Sommo, Alexander K. Mayer, 
Timothy Rudd, and Dan Cullinan, Commencement Day: Six-Year Effects of a Freshman Learning Community 
Program at Kingsborough Community College (New York: MDRC, 2012). 
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CUNY has conducted internal quasi-experimental analyses that provide evidence of 
CUNY Start’s effectiveness.* Building on that evidence base, MDRC, CUNY, and the Commu- 
nity College Research Center at Teachers College, Columbia University, are partnering to evalu- 
ate CUNY Start at four CUNY community colleges using a random assignment research design, 
supported by a grant from the federal Institute of Education Sciences. Each eligible student who 
consented to participate was assigned at random either to the program group, whose members 
could participate in CUNY Start, or to the control group, whose members could receive standard 
college courses and services, including standard developmental education courses. The difference 
between the two groups’ average outcomes provides an unbiased estimate of the effect of the 
program. 


This report is the first to share findings from the evaluation. Overall, the evaluation found 
that CUNY Start was implemented as it was designed and that there was considerable contrast 
between the program and the colleges’ standard courses and services. During the first semester 
of the study, program group students made much more progress through developmental education 
than control group students, while control group students earned more college credits, as predicted 
by CUNY Start’s designers. In the second semester, program group students enrolled at CUNY 
colleges (that is, participated in CUNY Start or enrolled in any non-CUNY Start courses as ma- 
triculated students) at a higher rate than control group students. 


CUNY Start’s Theory of Change 


CUNY Start’s underlying theory of change posits that students with substantial developmental 
course requirements are best served through an intensive model designed to build their academic 
preparedness and college skills before they matriculate. The program is designed to make students 
more engaged in their course work, help them view themselves as competent learners, give them 
the support they need to succeed, and prepare them for college-level work. The program’s low 
cost to students is expected to make it easier for them to participate. Because financial aid cannot 
be used to pay for CUNY Start, students can preserve their financial aid for future courses (de- 
velopmental education courses and college-level courses).* 


3Allen and Horenstein compare the outcomes of students in CUNY Start with a matched comparison group 
of students who did not enroll in CUNY Start but were similar with respect to their measurable characteristics 
(such as their number of developmental requirements). Such analyses rely on the assumption that after matching 
on measured characteristics, the students in CUNY Start and the comparison group also were similar with respect 
to their unmeasured characteristics that are related to the outcomes of interest (for example, tenacity and motiva- 
tion). See Drew Allen and Aaron Horenstein, CUNY Start: Analysis of Student Outcomes (New York: City Uni- 
versity of New York, 2013). 


4Students are eligible to receive federal Pell Grants for only six years and New York State Tuition Assistance 
Program grants for up to eight semesters, with a maximum of six semesters at the associate’s degree level. 
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The program’s designers hypothesize that compared with students in standard college 
courses (including standard developmental education courses), a higher proportion of CUNY 
Start students will complete developmental education and that they will do so more quickly. Be- 
cause CUNY Start students spend a semester building their basic skills before matriculating, they 
are expected to earn fewer college credits in the short term. Over the longer term, the expectation 
is that CUNY Start students will have higher retention rates (that is, more of them will stay in 
college), will accumulate more college-level credits, and will eventually have higher graduation 
rates. 


Evaluation Sample Members 


CUNY Start is available to incoming students who have been assessed as requiring remediation. ° 
It offers a full-time program and a part-time program. The full-time program is only open to stu- 
dents who are assessed as needing remediation in math and at least one other subject area (read- 
ing, writing, or both). The part-time program, which provides instruction in math or reading and 
writing, is open to students who are assessed as needing remediation in at least one subject area. 
Although any student who requires remediation is eligible for CUNY Start (and was eligible for 
the evaluation), CUNY Start staff members focus on recruiting a narrower population of students: 
those who have been assessed as needing remediation in all three subject areas. 


Students were randomly assigned to the program or control group before each of three 
semesters: spring 2015, fall 2015, and spring 2016. A total of 3,835 students were assigned. Stu- 
dents completed a questionnaire before they were randomly assigned, and that questionnaire 
shows that the research sample, like the broader student body at the colleges, is racially diverse, 
with many students whose native language is not English. Most students in the sample are 
women. Most reported that they lived with their parents, and very few said they had children. 
Most of the sample members were assessed as needing remediation in two or three subject areas, 
reflecting CUNY Start’s efforts to recruit such students. 


CUNY Start Model and Implementation 


From spring 2015 to spring 2016, the evaluation collected information on the implementation of 
CUNY Start and the standard offerings at the four colleges using several data sources, including 
interviews with administrators and instructors, observations of classrooms, and surveys of stu- 
dents and instructors. Table ES.1 compares aspects of the program with standard college courses 
and services. The elements of administration, cost, and structure shown in the table are fixed and 


The program also admits a small number of students with some limited college experience (typically fewer 
than 12 college credits). 
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Table ES.1 


Highlights of CUNY Start and Standard College Courses and Services 


Component 


CUNY Start 


Standard College Courses and Services 


Administration, cost 
and structure 


Administration 


Cost to student 


Structure 


Developmental math 
instructional approach 


Curriculum 


Pedagogy 


Developmental 
reading/writing 
instructional approach 


Curriculum 


Pedagogy 


Student support 
College success seminar 


Advising 


Tutoring 


Situated in Continuing Education; 
managed centrally 


$75; students not eligible for financial aid 


1 semester of developmental math, 
reading, and writing; students cannot take 
college level courses that semester; up to 
26.5 hours of instruction per week in the 
full-time program 


Arithmetic and algebra integrated; 
problems emphasize conceptual 
understanding; assignments include 
activities that develop students’ academic 
skills 


Mostly student-centered instruction 


Reading/writing content integrated; writing 
assignments designed to help students 
process and respond to reading material 


Mostly student-centered instruction 


Mandatory; most students take a seminar 


Student-to-adviser ratio 75:1; most 
surveyed students reported at least one 
one-on-one advising session in the past 
semester 


Almost half of surveyed students reported 
receiving tutoring 
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Situated in the Academic Affairs division; 
managed within academic departments 


Full-time tuition $2,400; students may be 
eligible for financial aid 


Up to 3 semesters of developmental math, 
reading, and writing; students can take 
selected college-level courses; typically 12- 
16 hours of instruction per week for a full- 
time student 


Arithmetic and algebra taught separately; 
academic skill-building activities not 
prevalent 


Mostly lecture-based instruction 


Reading/writing content typically not 
integrated; writing assignments in upper- 
level courses include research synthesis 
papers 


Mostly student-centered instruction 


Typically not mandatory; some students 
take a seminar 


Student-to-adviser ratio 600:1; many 
surveyed students reported at least one 
one-on-one advising session in the past 
semester 


Approximately one-third of surveyed 
students reported receiving tutoring 


(continued) 


Table ES.1 (continued) 


Instructor hiring and 


training 

Hiring Instructors hired based on content and Instructors typically hired based on content 
pedagogical knowledge and openness to knowledge 
CUNY Start instructional approach 

Training Most instructors participated in an Most instructors did not participate in 
apprenticeship; continuing professional training before teaching a course; 
development was regular and common continuing professional development was 


common but less regular and intensive 


SOURCES: Community College Research Center field research data; MDRC calculations using data from the 
instructor and student surveys; www2.cuny.edu/academics/academic-programs/model-programs/cuny-college- 
transition-programs/cuny-start; discussions with CUNY staff members. 


were not explored in the implementation research, but they are included here to provide a com- 
prehensive summary of the program. Overall, CUNY Start was implemented with fidelity to the 
program model, and there was a substantial contrast between the program and the colleges’ stand- 
ard courses and services, including their standard developmental education courses. 


Administration, Cost, and Structure 


CUNY Start is situated in each college’s Continuing Education division, which offers a 
range of courses outside the academic departments, including job-skills courses and adult basic 
education. Administrators in CUNY’s Office of Academic Affairs manage the program and pro- 
fessional development staff members provide training to CUNY Start instructors and advisers 
and develop program curricula. The CUNY Office of Academic Affairs works closely with 
CUNY Start directors at each college to operate the program, and provides administrative and 
evaluation support to ensure the program is well implemented and to measure its efficacy. Stand- 
ard developmental education and college-level courses are situated in the colleges’ academic de- 
partments, which are part of each college’s Office of Academic Affairs. 


As noted earlier, CUNY Start students pay only $75 for the semester, including the course 
materials. They cannot use financial aid, which allows them to retain the aid for the future. In 
contrast, tuition and fees for full-time students at the four study colleges in fall 2015 was $2,400 
(for New York State residents), and many students were eligible for financial aid. 


The structure of CUNY Start is unique. Its full-time program provides up to 26.5 hours 
of instruction per week during its one semester: 12 hours of math, 12 hours of integrated reading 
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and writing, and 1.25 to 2.5 hours in the college success seminar.® CUNY Start’s part-time pro- 
gram provides 12 hours of instruction in either math or reading and writing, and 1.5 hours in the 
seminar. Students cannot take college-level courses during that semester. In contrast, students 
who are not in CUNY Start might take multiple developmental education courses over multiple 
semesters, and each course typically meets 3 to 6 hours per week. Students can take some college 
courses at the same time, and a full-time student usually receives 12 to 16 hours of instruction per 
week (including developmental education and college-level courses). CUNY Start’s more inten- 
sive instruction is intended to help students satisfy their developmental education requirements in 
one semester. 


Math and Reading/Writing Instructional Approach 


CUNY Start math and reading/writing instructors use prescribed curricula that were cre- 
ated by experienced faculty members and CUNY Start professional development staff members 
for program-wide use. Program curricula are regularly refined by the professional development 
staff based on comments from the instructors. 


CUNY Start’s math instructional approach is markedly different from standard develop- 
mental math instruction. CUNY Start integrates arithmetic and algebra and encourages concep- 
tual understanding, real-world learning, and the building of academic skills such as studying and 
note taking. It relies primarily on “student-centered” instruction: Instructors facilitate meaningful 
student discussion and engagement with the material, and frequently ask specific, open-ended 
questions to stimulate student discussion. Instructors in standard remedial math classes tend to 
use more lecturing. 


CUNY Start teaches reading and writing integrated in one class, unlike many develop- 
mental courses, to reinforce the relationship between the two and to allow students to move more 
quickly through their developmental requirements. Unlike standard developmental reading and 
writing instruction, the CUNY Start instructional approach draws on the “cognitive apprentice- 
ship” model, in which instructors aim to help students learn the habits and techniques of proficient 
readers and writers.” In both CUNY Start and standard developmental reading and writing, in- 
structors rely on student-centered instruction, so the pedagogical difference between CUNY and 
non-CUNY Start instruction is less substantial in reading and writing than it is in math. 


SAt most colleges, the college success seminar for full-time students meets for 2.5 hours for the first four 
sessions, and then 1.25 hours for the remaining sessions; however, some campuses offer a consistent 1.25- to 
1.5-hour seminar for all full-time students. 


7Allan Collins, John Seely Brown, and Susan E. Newman, Cognitive Apprenticeship: Teaching the Craft of 
Reading, Writing, and Mathematics (Champaign, IL: Center for the Study of Reading, University of Illinois at 
Urbana-Champaign, 1987). 
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Student Support 


CUNY Start’s student support is more integrated into the classroom than is typical at 
community colleges. CUNY Start’s mandatory college success seminar, led by a CUNY Start 
adviser, aims to help students develop skills to balance school and life, solve problems, advocate 
for themselves, and view themselves as learners. Typically, non-CUNY Start students are not 
required to participate in a college success seminar. CUNY Start advisers, who are responsible 
for far fewer students than non-CUNY Start advisers, also meet with students outside the seminar 
to give them support during the program and to plan for their matriculation. CUNY Start math 
tutors and writing assistants provide help to students inside and outside of class. When surveyed, 
program group students were somewhat more likely than control group students to report that 
they had met with an adviser or a tutor outside of class. 


Hiring and Training 


Typically, college instructors are hired primarily based on their content knowledge (and 
of course their academic credentials). CUNY Start hires instructors based on their content and 
pedagogical knowledge, and their openness to the prescribed curriculum and pedagogy. Once 
hired, CUNY Start instructors are expected to participate in a semester of apprenticeship before 
they teach their own classes, during which time they observe and assist experienced instructors. 
After that semester, instructors continue to receive training. Most CUNY Start instructors sur- 
veyed for the study had participated in apprenticeships and almost all reported receiving com- 
ments on their instruction. In contrast, most non-CUNY Start developmental education instruc- 
tors who were interviewed did not report participating in training before teaching a course for the 
first time. Many reported receiving some kind of professional development, but they participated 
for fewer hours than CUNY Start instructors. 


The Effects of CUNY Start 


e During the first semester in the study, program group students made sub- 
stantially more progress through developmental education than control 
group students, while control group students earned more college credits. 


This finding reflects CUNY Start’s focus on reducing or eliminating developmental 
course requirements before students matriculate and is in line with the program’s theory of 
change. As Table ES.2 shows, before random assignment similar percentages of program group 
students and control group students were “college-ready” in math, reading, and writing (accord- 
ing to their scores on placement tests, primarily). By the end of their first semester in the study 
(the “program semester”) more program group students than control group students were college- 
ready in each of the subject areas (as demonstrated by their scores on exit tests or their completion 
of the highest level developmental education course in that subject area). The difference is largest 
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Table ES.2 
Effects on Educational Achievement 


Program Control 


Outcome Group Group Difference P-Value 


Before random assignment 
College-ready in the subject area (%) 


Math 5.4 5.8 -0.3 0.704 
Reading 36.6 35.7 0.9 0.642 
Writing 22.3 23.6 -1.3 0.428 


End of the program semester 
College-ready in the subject area (%) 


Math 56.8 24.7 32.1 *** <0,001 

Reading 69.7 61.6 8.0 *** <0.001 

Writing 61.0 51.6 94 *** <0.001 
College-ready in all three subject areas (%) 37.9 13.0 24.9 *** <0.001 
College-level credits earned 0.6 2.4 -19 ** <0.001 
Sample size (total = 3,835) 2,997 838 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY's Institutional Research Database, and test data from CUNY's Administrative Data 
Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random 
assignment blocks. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. See 
Appendix E for details on the impact-estimation model. 


in math: By the end of the program semester, 57 percent of program group students were college- 
ready in math, compared with 25 percent of control group students. By the end of the program 
semester, 38 percent of program group students were college-ready in all three subject areas, 
compared with 13 percent of control group students. Using transcript data from CUNY, Table 
ES.2 also shows that program group students earned fewer college credits than control group 
students in the program semester. This result is expected, since CUNY Start students had not 
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matriculated and therefore could not earn college-level credits, while control group students had 
matriculated and did enroll in some college-level courses.® 


e During the second semester, program group students enrolled at CUNY 
colleges (that is, participated in CUNY Start or enrolled in any non- 
CUNY Start courses as matriculated students) at a higher rate than con- 
trol group students. 


Sixty-nine percent of the program group enrolled at CUNY colleges during the second 
semester of the study, compared with 64 percent of the control group. This difference mostly 
reflects the fact that program group students participated in CUNY Start at a higher rate than 
control group students. Similar percentages of the two groups enrolled in any non-CUNY Start 
courses. (These findings are not shown in a table.) 


e Ineach of the subgroups of students examined, the program group made 
more progress in developmental education and earned fewer college cred- 
its in the program semester than the control group. 


The findings for the full research sample represent the program’s effects on average. Dif- 
ferent types of students, however, may respond differently to CUNY Start or the services availa- 
ble to the control group, and thus, the effects may vary among different groups of students. Sev- 
eral different subgroups of students were examined, including: those who intended to participate 
in CUNY Start full time or part time; those who required remediation in all three subjects and 
those who required remediation in only one or two; those at each of the four colleges in the study; 
those of various races/ethnicities; and men and women. Analyses show that CUNY Start had 
positive effects on progress through developmental education and negative effects on college 
credits earned in the program semester for all groups examined. 


Discussion of Findings and Looking Forward 


Within one semester, CUNY Start enabled students to make substantial progress through 
developmental education — more progress than has been seen in most other developmental edu- 
cation reforms that have been evaluated in random assignment studies.° The program is meeting 


®The report includes findings on enrolling at CUNY colleges in the program semester, defined as participat- 
ing in CUNY Start or enrolling in any non-CUNY Start courses as a matriculated student. A higher proportion 
of program group students than control group students enrolled at CUNY colleges in that semester. An analysis 
described in the report strongly suggests that the effects reported here on completing developmental education 
and accumulating college credits in the program semester are not simply the result of this enrollment effect. 

°See, for example, A.W. Logue, Mari Watanabe-Rose, and Daniel Douglas, “Should Students Assessed as 
Needing Remedial Mathematics Take College-Level Quantitative Courses Instead? A Randomized Controlled 
Trial” (Educational Evaluation and Policy Analysis 38, 3: 578-598, 2016); Susan Scrivener, Michael J. Weiss, 
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its goal of helping students substantially reduce or complete their developmental education re- 
quirements within a semester. The effects in math are especially striking, since developmental 
math is a barrier that prevents many students from earning a degree.'° CUNY Start’s short-term 
success is also striking given that the program targets students with substantial developmental 
course requirements, in contrast to many other reforms.'! By enabling students to make substan- 
tial progress in or complete their developmental education requirements, CUNY Start can help 
students maintain their future financial aid eligibility — repeating developmental courses, which 
is acommon occurrence for students taking standard developmental education courses, can affect 
students’ ability to make “satisfactory academic progress” for continued aid eligibility.!* The 
greater progress students make through developmental education can also help them avoid reach- 
ing the limits on their aid. 


The positive early results in this report are only part of the story. It is essential to learn 
how the trade-off of making a short-term priority of developmental education rather than college- 
level credit accumulation will play out in the longer term. Additional follow-up data in this eval- 
uation will provide information about sample members’ persistence in college, college credit ac- 
cumulation, and graduation rates. If CUNY Start’s short-term trade-off results in the hypothesized 
longer-term gains, the program will serve as an important model for serving students with sub- 
stantial developmental education requirements. 


The research team will track the academic progress of students in the study for at least 
two years after they were randomly assigned and will examine the program’s costs. A report on 
longer-term effects and cost-effectiveness is scheduled to be published by MDRC in 2020. In 
addition, the Community College Research Center will publish two papers focused on CUNY 
Start’s math curriculum and pedagogy and its staffing and professional development model. Fi- 
nally, CUNY will develop a tool kit on CUNY Start implementation and best practices, focusing 
on CUNY’s use of data to inform program management and refinement. 


Alyssa Ratledge, Timothy Rudd, Colleen Sommo, and Hannah Fresques, Doubling Graduation Rates: Three- 
Year Effects of CUNY’s Accelerated Study in Associate Programs (ASAP) for Developmental Education Students 
(New York: MDRC, 2015); and Elizabeth Zachry Rutschow and Emily Schneider, Unlocking the Gate: What 
We Know About Improving Developmental Education (New York: MDRC, 2011). 


10Paul Attewell, David Lavin, Thurston Domina, and Tania Levey, “New Evidence on College Remedia- 
tion” (Journal of Higher Education 77, 5: 886-924, 2006). 

117 ogue, Watanabe-Rose, and Douglas (2016). 

12Satisfactory academic progress requirements vary from college to college and can include not attempting 
too many credits (generally more than 150 percent of the credits required for a degree), maintaining a cumulative 
grade point average of 2.0 or higher, completing at least two-thirds of credits attempted, and, at some colleges, 
completing a certain number of credits by the end of each year. See Judith Scott-Clayton and Lauren Schudde, 
“Performance Requirements in Need-Based Aid: What Roles Do They Serve, and Do They Work?” (New York: 
Center for Analysis of Postsecondary Education and Employment, Teachers College, Columbia University, 
2017); https://studentaid.ed.gov/sa/eligibility/staying-eligible. 
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Chapter 1 


Introduction 


Many students who enter community college are deemed academically underprepared for 
college-level courses and are referred to developmental (remedial) education courses to build 
their math, reading, or writing skills.! These students often struggle in the developmental courses 
and in college more broadly. Graduation rates for such students are low, especially among those 
who are assessed as needing remediation in multiple subjects. 


Hoping to boost the success rates of its least prepared incoming students, the City Uni- 
versity of New York (CUNY) developed CUNY Start, an innovative developmental education 
program. CUNY Start’s full-time program was designed for and targets incoming students who 
are assessed as needing remediation in math, reading, and writing (based on their scores on the 
CUNY Assessment Tests, or “placement tests’).? It is also open to students who are assessed as 
needing remediation in math and either reading or writing.* The program’s short-term goal is to 
substantially reduce or eliminate students’ developmental education course requirements after 
one semester, while preparing them for college-level courses. Its long-term goal is to improve 
academic outcomes, including graduation rates. CUNY Start delays college matriculation for one 
semester and provides instruction in math, reading, and writing over a compressed period with a 
prescribed curriculum and instructional approach.° It also provides advising and tutoring and a 
weekly seminar that teaches students skills they need to succeed in college. Students pay only 
$75 for the program, including the course materials. 


MDRC, CUNY, and the Community College Research Center at Teachers College, Co- 
lumbia University, are partnering to evaluate CUNY Start, supported by the federal Institute of 
Education Sciences. The evaluation examines how CUNY Start is implemented, its costs, and its 
effects on academic outcomes. To estimate the program’s effects on students’ academic out- 
comes, the evaluation uses a random assignment design to compare students in a program group, 
whose members had access to CUNY Start, with students in a control group, whose members had 


‘Bailey, Jeong, and Cho (2010). 

*See, for example, Attewell, Lavin, Domina, and Levey (2006); Bailey, Jeong, and Cho (2010); and Figure 
1.1. 

3As is described in Appendix B, CUNY community colleges require incoming students to take the CUNY 
Assessment Tests if they have not demonstrated college-level proficiency in math, reading, or writing through 
their scores on the SAT or New York State Regents exams (statewide standardized tests in core high school 
subjects). 

4As is described later in the chapter, CUNY Start also has a part-time program for students with one or more 
developmental education requirements. 

°In this report “matriculation” is when a student first enrolls in non-CUNY Start courses in a degree program. 


access to the standard courses and services at CUNY’s community colleges, including the stand- 
ard developmental education courses (everything other than CUNY Start). 


This report is the first to share findings from the evaluation; it includes implementation 
findings and effects on early academic outcomes. As the report discusses in detail, the colleges in 
the study generally implemented CUNY Start as it was designed, and there were substantial dif- 
ferences between CUNY Start and the colleges’ standard courses and services. After a semester, 
program group students made much more progress through their developmental education re- 
quirements than control group students, while control group students earned more college credits 
— in line with CUNY Start’s theory of change, which is described below. During sample mem- 
bers’ second semester in the study, program group students enrolled at CUNY colleges (that is, 
participated in CUNY Start as nonmatriculated students or enrolled in non-CUNY Start courses 
as matriculated students) at a higher rate than control group students. Additional semesters of 
follow-up data, which will be presented in a future report, will determine the degree to which 
CUNY Start’s short-term focus on reducing or eliminating developmental education require- 
ments yields greater longer-term academic success. 


Developmental Education Nationally and at CUNY 


Nearly half of all undergraduates in the United States attend community colleges. A dispropor- 
tionate number of them are low-income students, underrepresented minorities, nontraditional stu- 
dents, and first-generation college students.° Community colleges have increased access to higher 
education not just because they are affordable but because they are open-access, meaning that 
anyone with a high school diploma or equivalency can matriculate. 


For as long as community colleges have been open-access, they have faced the challenge 
of students who arrive and are assessed as being underprepared for college-level work. Typically, 
these students are referred to developmental education, where they are required to complete one 
or more noncredit math, reading, or writing courses before enrolling in college-level, credit- 
bearing math and English courses, and sometimes other college-level courses. (For example, 
students who require developmental math are often prohibited from enrolling in college-level 
science courses.) 


°Berkner and Choy (2008); Provasnik and Planty (2008); Shapiro et al. (2017). First-generation students are 
those whose parents did not attend college. Nontraditional students include those who have delayed their enroll- 
ment in college (that is, who did not enter college in the same calendar year as they finished high school), who 
are attending part time, who are working full time, who are financially independent from their parents, who have 
dependents, and who are single parents. 


Today, many students entering community colleges are referred to some developmental 
education.’ But taking noncredit course sequences slows down students’ progress toward gradu- 
ation and leads them to incur additional tuition costs, and in the process, expend a portion of their 
financial aid.* Furthermore, taking (and often repeating) noncredit course sequences can put stu- 
dents at risk of failing to meet satisfactory academic progress (SAP) requirements for maintaining 
federal aid eligibility.° Federal aid also has time limits: Students are only eligible to receive the 
equivalent of six years of Pell Grant funding.'° The additional time and aid required to complete 
developmental education requirements are clearly a high cost, since many students do not finish 
their assigned developmental course sequences, let alone graduate. "! 


Graduation rates from community colleges are quite low overall, but they are even lower 
among students who are referred to developmental education. Figure 1.1 shows graduation rates 
for full-time, first-time students.'? Nationwide, 31 percent of those students graduated (that is, 
earned associate’s degrees) within three years. However, the three-year graduation rate among 
the subset of students who took at least one developmental course is less than half that: 13 per- 
cent.'? Like other urban community colleges, CUNY’s community colleges have a lower overall 
graduation rate than the national average: 17 percent of first-time, full-time students graduated 
within three years. At CUNY, 14 percent of students who were assessed as needing at least one 
developmental course graduated within three years. Not surprisingly, graduation rates are even 
lower among students who require remediation in all three subjects (math, reading, and writing). 
At CUNY’s community colleges, only 7 percent of the students who entered in fall 2011 and who 
were assessed as needing remediation in math, reading, and writing (“triple remedial” in Figure 
1.1) graduated within three years. 


"Bailey, Jeong, and Cho (2010). 

8CUNY Task Force on Developmental Education (2016). 

°SAP requirements vary from college to college and can include not attempting too many credits (generally 
more than 150 percent of the credits required for a degree), maintaining a cumulative grade point average of 2.0 
or higher, completing at least two-thirds of credits attempted, and, at some colleges, completing a certain number 
of credits by the end of each year. See Scott-Clayton and Schudde (2017); City University of New York (2018c); 
and U.S. Department of Education, Office of Federal Student Aid, “Staying Eligible” (n.d.). 

10See U.S. Department of Education, Office of Federal Student Aid, “Calculating Pell Grant Lifetime Eli- 
gibility Used” (n.d.). 

'lRewer than 30 percent of students who place into the lowest levels of reading and math ever complete their 
required course sequences. See Bailey, Jeong, and Cho (2010) and Scott-Clayton, Crosta, and Belfield (2014). 

!Due to the availability of data, these data represent students entering college in fall 2011. 

13This percentage does not include students who were assessed as needing developmental courses but never 
took them, and thus never graduated. 


Figure 1.1 
Community College Three-Year Graduation Rates 
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SOURCES: MDRC calculations based on data from CUNY; Ginder, Kelly-Reid, and Mann (2015); U.S. 
Department of Education, National Center for Data Statistics (n.d.); and City University of New York Office of 
Institutional Research and Assessment (2015). 


NOTES: These statistics represent full-time, first-time students who entered two-year institutions in fall 2011. For 
community colleges (nationwide), "developmental students" are those who took a developmental course. 
"Developmental students" at CUNY community colleges are those who were assessed as needing at least one 
developmental course based on their initial skills proficiency scores on the CUNY Assessment Tests in math, 
reading, and writing. "Triple-remedial students" are CUNY students who did not pass any of the CUNY 
Assessment Tests in math, reading, and writing. 


The low rate of success for students assessed as needing developmental education has 
prompted many colleges and states to redesign their developmental offerings. Most commonly, 
reform approaches have focused on changing developmental courses’ structure or timing (or 
both), changing their curricula, changing how they are taught, changing how students are assessed 
and placed into courses, and providing additional support for developmental education students. 


Many reforms have been tried. Some have yielded improved outcomes for students in the short 
term, but most have not substantially affected their long-term outcomes. '* 


As at most community colleges, developmental education at CUNY has typically com- 
prised multilevel, multisemester, noncredit course sequences in math, reading, and writing. But 
as reforms have been explored nationwide, CUNY’s leaders have also focused on rethinking the 
university’s approaches, and CUNY and its colleges have implemented many different changes. 
As this report was being written, CUNY was working to revamp its developmental education, 
introducing alterations in how students are assessed and placed in courses, how students qualify 
to move out of developmental education, and the content and pedagogy of the courses themselves. 
The shifts draw on previous reforms tried at individual CUNY institutions and across CUNY as 
a whole.'> CUNY Start is continuing to operate in the midst of these other reforms. 


CUNY Start incorporates most of the reform approaches mentioned above — including 
changes to structure and timing, curriculum, pedagogy, and student support — and is uncom- 
monly multifaceted. CUNY Start targets students who require remediation in math, reading, and 
writing. This focus on “triple-remedial” students makes the program and its evaluation unique 
among the existing body of work on developmental education reform, much of which has focused 
on students who have developmental requirements in only one or two subject areas, or those who 
are on the cusp of being deemed “college-ready.” '* Additionally, unlike most other developmen- 
tal reform approaches, CUNY Start aims to provide all the developmental education students 
need before they matriculate. There are other prematriculation initiatives designed to support ac- 
ademically underprepared students, such as bridge programs, “boot camps,” and some Massive 
Open Online Courses, but programs that postpone matriculation — like CUNY Start — are un- 
common.'’ CUNY Start also costs students very little: Students pay $75 to participate, a sum that 
includes the cost of materials. CUNY has conducted internal analyses that provide evidence of 
CUNY Start’s effectiveness, and has expanded the program in part based on those results. '® The 


'47achry Rutschow and Schneider (2011); Adelman (2004); Attewell, Lavin, Domina, and Levey (2006); 
Jenkins, Jaggars, and Roksa (2009); Bailey, Jeong, and Cho (2010). CUNY’s Accelerated Study in Associate 
Programs (ASAP) is an exception. The unusually comprehensive program, which provides up to three years of 
financial and academic support and other forms of support for students, almost doubles three-year graduation 
rates. ASAP targets students with up to two developmental education requirements. For more information, see 
Scrivener et al. (2015) and Strumbos and Kolenovic (2016). 

'SSee CUNY Task Force on Developmental Education (2016). 


16Hodara and Jaggars (2014); Edgecombe, Jaggars, Xu, and Barragan (2014); and Sommo, Mayer, Rudd, 
and Cullinan (2012). 

‘7Students in summer bridge programs and developmental education “boot camps” receive accelerated de- 
velopmental education instruction before their first semester in college. Using Massive Open Online Courses, 
students can receive accelerated developmental education instruction online before they matriculate. For exam- 
ples of other prematriculation programs, see California State University (2011) and City University of New York 
(2018b). 


18 Allen and Horenstein (2013). 


present study builds on the existing empirical evidence base, offering an independent random 
assignment evaluation of the program’s effects. 


The CUNY Start Model 


CUNY Start began in 2009 as a small, intensive program that targeted students with General 
Educational Development (GED) certificates (now high school equivalency diplomas) who had 
been assessed as having very weak math, reading, and writing skills. It expanded in 2010 to serve 
high school graduates also. It currently operates at six of CUNY’s seven community colleges, the 
College of Staten Island, and Medgar Evers College. Program enrollment has grown greatly over 
time. In 2009, CUNY Start served about 150 students; by fall 2017, it had served a total of 18,000 
students with an annual enrollment of 3,500. 


The program is situated in each college’s Continuing Education division, which offers a 
range of courses outside the academic departments, including job-skills courses and adult basic 
education. Administrators in CUNY’s central Office of Academic Affairs manage the program 
and a group of professional development staff members provides training and support to CUNY 
Start instructors and advisers, and develops program curricula. The Office of Academic Affairs 
works closely with CUNY Start directors and program coordinators at each college to oversee 
and implement the program, and provides evaluation support to measure the program’s efficacy. 
The colleges’ CUNY Start teams also include math and reading/writing instructors, advisers, 
math tutors, and writing assistants. 


CUNY Start’s underlying theory of change posits that students with substantial develop- 
mental education requirements are best served through an intensive, cohort-based model (in 
which a group of students who join at the same time take all of their classes together) designed to 
build academic preparedness and college skills before matriculation. The program’s components 
are designed to make students more engaged in their course work, help them view themselves as 
learners, help them make greater connections with peers, give them the support they need to suc- 
ceed, and prepare them for college-level work. The program’s low cost to students is expected to 
make it easier for them to participate. It also allows them to preserve their financial aid for future 
courses, since they do not tap those funds for CUNY Start. 


The program’s designers hypothesize that compared with students in standard college 
courses, a higher proportion of CUNY Start students will build their basic skills and complete 
developmental education, and that they will do so more quickly. Because CUNY Start students 
spend a semester building their basic skills before matriculating, they are expected to earn fewer 
college credits in the short term. The hypothesis, however, is that over the longer term (beginning 
in students’ fourth semester), CUNY Start students will have higher retention rates (that is, more 


of them will stay in college), will accumulate more college-level credits, and will eventually have 
higher graduation rates. 


The rest of this section describes CUNY Start’s components and provides more infor- 
mation on why the program might improve students’ outcomes. The section describes the CUNY 
Start model as it was designed; Chapter 2 describes how the model was implemented in practice 
at the colleges in the evaluation. 


Structure 


Typically, as described above, students who are referred to developmental education are 
required to complete one or more noncredit math, reading, or writing courses before they can 
enroll in college-level math or English courses for which they can earn college credits. Each de- 
velopmental course typically meets three to six hours per week. Completing these courses may 
take several semesters, and students can take some college-level courses at the same time. Stu- 
dents pay regular undergraduate tuition for both their developmental and college-level courses 
(which was, for example, $2,400 for a New York State resident attending a CUNY community 
college full time in the fall 2015 semester), and they may be eligible for financial aid. 


CUNY Start attempts to prepare students for college-level math and English in one 
semester. In CUNY Start’s full-time program, students take classes in math and integrated 
reading/writing for 24 hours each week. In the part-time program, students take either math or 
reading/writing for 12 hours each week. All CUNY Start students also spend time each week in 
a college success seminar.*° CUNY Start students spend considerably more time per week in math 
and reading/writing courses than students in traditional developmental courses typically do. 
This instructional intensity is intended to increase the amount of time that students spend actively 
engaged with learning activities and facilitates the use of the CUNY Start curriculum and 
pedagogy discussed below. Evaluations of both community college and high school programs 
and of policies that significantly increase the time spent in targeted subject areas have shown that 


'9 Alexandra Logue and CUNY Office of Institutional Research and Assessment, email to author, February 
6, 2018. 


°° At most colleges, the seminar for full-time students meets for 2.5 hours for the first four sessions, and then 
1.25 hours for the remaining sessions. Some campuses, however, offer a consistent 1.25- to 1.5-hour seminar for 
all full-time students. The seminar for part-time students meets for 1.5 hours. 

1 Across semesters, the total time students spend in standard developmental courses varies depending on 
the number of courses they need to take, how many hours each of those courses meet, and whether the students 
pass the courses or need to retake them. 


this approach can help students succeed in those subject areas.** As noted earlier, during their 
time in CUNY Start, students do not matriculate at college or take any college-level courses. 


CUNY Start uses a cohort model, in which students participate as a group in math, 
reading/writing, and the college success seminar. Such grouping is hypothesized to foster stronger 
connections among students and between students and instructors,’ and those connections are 
positively associated with persistence in college.*4 Past experimental research provides evidence 
that “learning communities” that enroll groups of students in clusters of classes can have a small 
positive effect on students’ academic outcomes.” 


After 12 weeks of CUNY Start — Phase I of the program — students take exit tests in 
the subject areas they have been studying.” Students who pass are eligible to take college-level 
courses the next semester. Students who do not pass receive an additional three to six weeks of 
CUNY Start instruction — Phase II — before being reassessed. If students are assessed as need- 
ing more remediation after Phase II, they cannot repeat CUNY Start, but they can matriculate at 
the college and take the standard developmental education courses (or free “immersion” courses 
in the summer or between terms to build math, reading, or writing skills), and they can take some 
college-level courses. If students take CUNY’s part-time program and still have developmental 
requirements when they are done, they can participate for an additional semester in the CUNY 
Start subject area(s) they did not take previously. For example, a student who participated in the 
part-time program in math could later participate in the part-time program in reading/writing. 


As mentioned above, CUNY Start students pay a flat fee of $75 for their courses and 
course materials.*”? They cannot use financial aid for the program, which ensures that they retain 
all aid for future courses, when they may be better prepared for college-level work and may be 
taking more credit-bearing courses. 


*2 Jenkins et al. (2010); Kemple, Herlihy, and Smith (2005). 

*3Visher, Wathington, Richburg-Hayes, and Schneider (2008). 

*4Tinto (1993); Astin (1993). 

*5Richburg-Hayes, Visher, and Bloom (2008); Weiss, Visher, Weissman, and Wathington (2015). 

?6 As is described in Appendix B, during the period covered in this report, the CUNY Assessment Tests were 
used for both placement into developmental education in all subject areas and as the exit tests for developmental 
reading and writing. The CUNY Elementary Algebra Final Exam was used as the exit test for developmental 
math. 

27CUNY is able to charge only $75 because of some differences between CUNY Start and standard devel- 
opmental education. Both result in reimbursement from New York State based on the number of students en- 
rolled and the number of hours of class time. However, hourly instructional expenses are somewhat lower for 
CUNY Start. The student fees and the reimbursement from the state cover close to 70 percent of the total cost 
per CUNY Start student, with the balance coming from CUNY’s general funds. 


Curriculum 


The curricula in developmental education courses vary greatly. Traditional developmen- 
tal courses sometimes seek to cover a broad range of topics, and competencies are sometimes 
taught in isolation, without clear connections to college-level performance requirements.”® 


CUNY Start’s prescriptive math and reading/writing curricula were originally developed 
by experienced faculty members and professional development staff members from CUNY’s 
Adult Literacy and Education programs. They emphasize a number of research-based practices. 
For example, national college-readiness math standards and research on math instruction for 
adults and older adolescents both suggest that math courses should focus on developing students’ 
math proficiencies (such as problem solving and adaptive reasoning) and conceptual understand- 
ing (that is, their comprehension of math concepts, operations, and relations).”° The CUNY Start 
math curriculum therefore focuses on math concepts in depth, rather than on memorization and 
mechanical repetition, and asks students to apply their newly gained skills in real-life situations. 
Such features are intended to show students how the subject matter is relevant to them.*° Doing 
so has been found to have positive effects on students’ learning.*! 


Similarly, the CUNY Start reading/writing curriculum draws on research suggesting that 
students should receive explicit instruction in reading comprehension and writing strategies that 
are modeled and “scaffolded” by instructors: Students are coached as they practice and internalize 
these strategies, taking on more and more responsibility for the strategies as the semester pro- 
gresses.** These practices also reflect national K-12 language arts standards that say teaching 
should help students develop their understanding of concepts, their ability to think critically, and 
their ability to apply what they learn in the real world. 


Pedagogy 


Although pedagogical approaches vary greatly across standard developmental education 
courses and instructors, these courses often rely on lectures to teach students. This traditional 


8Grubb (2010); Barragan and Cormier (2013); Hern (2010). 


°U.S. Department of Education, Office of Vocational and Adult Education (2011); National Governors 
Association Center for Best Practices, Council of Chief State School Officers (2010); National Mathematics 
Advisory Panel (2008); National Council for Teachers of Mathematics (2000). 


3°Perin (2013). 
31Stein, Smith, Henningsen, and Silver (2009); Hiebert et al. (2003). 


32Graham and Perin (2007a, 2007b); Graham and Hebert (2010); National Institute of Child Health and 
Human Development (2000). 


33National Governors Association Center for Best Practices, Council of Chief State School Officers (2010). 


“remedial pedagogy,” characterized by lecture, rote, and procedural learning, positions students 
primarily as recipients of information. 


CUNY Start, in contrast, intentionally positions students as active learners by having in- 
structors guide the learning process rather than lecture. As advocated in recent K-12 math and 
language arts standards, *° CUNY Start instructors pose questions that prompt students to explore, 
explain, and discuss ideas with each other. This approach encourages students to apply the 
knowledge and skills modeled by instructors, and allows them to develop their own conclusions 
about the material presented. CUNY Start expects that students will be more engaged than they 
would be in classrooms with traditional pedagogy, and that they will develop knowledge they can 
use in future courses, stronger academic identities, and increased commitment to college. Indeed, 
past research suggests these teaching approaches play an important role in improving students’ 
achievement, and has found correlations between such engagement and academic success.°° 


Student Support 


Except in specialized programs, students in developmental education courses are not gen- 
erally required to avail themselves of any specific academic and nonacademic support services, 
nor are those services integrated with students’ course work.*” 


In contrast, CUNY Start includes a set of coordinated academic and nonacademic support 
services. A mandatory college success seminar is designed to help students develop skills and 
dispositions to help them balance school and life commitments, solve problems, advocate for 
themselves, and view themselves as learners. The seminar is led by an adviser who also provides 
one-on-one advising to participating students. CUNY Start typically has student-to-adviser ratios 
of approximately 75 to 1, substantially smaller than the ratios usually found at community 
colleges. The relatively small caseload allows CUNY Start advisers to provide more intensive 
assistance to students with the most needs.** In addition, advisers regularly meet with math and 
reading/writing instructors to discuss students’ successes and challenges. As part of this team 
approach, advisers and instructors meet with individual students mid-semester to discuss their 
progress. Past experimental evidence indicates that increased advising can modestly improve 


4Grubb (2010); Hodara (2011). 


35See, for example, National Governors Association Center for Best Practices, Council of Chief State School 
Officers (2010). 


367,achry Rutschow and Schneider (2011); McClenney, Marti, and Adkins (2012). 
3’Bailey, Jaggars, and Jenkins (2015). 
38K arp (2013). 
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student outcomes.*? Finally, CUNY Start math tutors and writing assistants provide academic 
support both inside and outside the classroom. 


Instructor Hiring and Training 


Typically, community colleges hire developmental education instructors based on their 
expertise in the subject matter they will be teaching (and of course their academic credentials). 
These instructors may have little experience working with students who require significant aca- 
demic and nonacademic support. Even though most have had some pedagogical training, com- 
munity colleges typically offer limited and optional professional development activities, few of 
which focus explicitly on instruction. In addition, college instructors are usually given a great deal 
of autonomy in how they instruct and support students. 


In contrast, CUNY Start administrators recruit potential math and reading/writing in- 
structors based on their content and pedagogical knowledge and their openness to the CUNY 
Start instructional approach. Once hired, CUNY Start instructors are expected to spend a paid 
apprenticeship semester observing and assisting lead instructors before they begin teaching their 
own classes. After the apprenticeship period, CUNY Start professional development staff mem- 
bers provide continuing training and support. Because CUNY Start is housed in Continuing Ed- 
ucation, where instructors typically do not have the same degree of autonomy as college instruc- 
tors, it can more easily use a standardized curriculum and pedagogical approach, and can require 
certain types of instructor training. (New CUNY Start advisers are also expected to participate in 
a semester-long apprenticeship and receive additional training from experienced CUNY Start ad- 
visers and professional development staff members.) 


A growing body of evidence on faculty development suggests that faculty members are 
more likely to adopt teaching methods if their professional learning opportunities are embedded 
in their work and are closely tied to their everyday practice.*° These approaches to recruitment 
and professional development are intended to ensure that CUNY Start students receive support 
from skilled instructors who understand the program’s curriculum and pedagogy and can imple- 
ment them. 


3°Scrivener and Weiss (2009). 


“Bragg and Bamett (2008); Edgecombe, Jaggars, Baker, and Bailey (2013); Bickerstaff and Edgecombe 
(2012). 
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Previous Research on CUNY Start’s Effects 


Before this evaluation began, CUNY conducted an internal, quasi-experimental analysis 
to estimate CUNY Start’s effects.*! The analysis compared the outcomes of students who partic- 
ipated in CUNY Start with the first-semester outcomes of a matched comparison group of stu- 
dents who did not participate in CUNY Start and who had similar measured demographic char- 
acteristics and levels of developmental requirements (requirements in math plus requirements in 
either reading or writing, or both). 


The analysis found that among CUNY Start students who required remediation in math, 
53 percent reached college-level proficiency in math by the end of the program, compared with 
10 percent (within a similar time period) of a comparison group of students who entered CUNY 
without first participating in CUNY Start. Fifty-seven percent of CUNY Start students who re- 
quired remediation in reading reached college-level proficiency in reading compared with 33 per- 
cent of a comparison group, and 62 percent of CUNY Start students who required remediation in 
writing reached college-level proficiency in writing compared with 26 percent of a comparison 
group. Further, once students who completed CUNY Start began degree programs, they at- 
tempted and earned more credits than comparison group students. Finally, by their fourth semes- 
ter, students who completed CUNY Start were more likely to still be in college than the compar- 
ison group students. 


This Evaluation 


Given the very positive findings from CUNY’s own research on CUNY Start, the program’s 
substantial enrollment, and its potential for significant growth, CUNY’s leaders wanted to obtain 
the most accurate evidence possible regarding CUNY Start’s effectiveness. They decided, there- 
fore, to enter into a partnership with MDRC and the Community College Research Center to 


41Quasi-experimental research methods estimate effects without randomly assigning participants to a pro- 
gram group or control group. CUNY’s internal quasi-experimental analyses used a technique called “propensity- 
score matching.” Such an analysis compares the outcomes of a group of students in a program (in this case, 
CUNY Start) with a comparison group of students with similar measured background characteristics. As in an 
experiment, the comparison group is intended to represent what would have happened to the program group had 
they not been in the program. The differences between the two groups are thus an estimate of the effect of the 
program. In an experiment where students are randomly assigned, the program group and control group are 
initially expected to be similar even in their unmeasured characteristics (for example, grit, motivation, or ability). 
When propensity-score matching is used, there is a risk that initially the program group and comparison group 
may not have similar unmeasured characteristics. Consequently, the differences in outcomes between program 
group and comparison group members found using propensity-score matching may be a result of the program, 
the result of the unmeasured initial differences between the two groups, or both. 


42 Allen and Horenstein (2013). 
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conduct a random assignment evaluation, widely considered the “gold standard” of evaluation 
design. 


Colleges in the Evaluation 


The CUNY system consists of 24 institutions, including community colleges, four-year 
colleges, and freestanding graduate and professional institutions; it serves over a quarter of a mil- 
lion matriculated students each semester.* In spring 2015, when the study began, CUNY Start 
was operating on seven CUNY campuses. (At the time this report was written, it was operating 
on eight.) Of those seven, the CUNY Start evaluation is taking place at four: Borough of Man- 
hattan Community College, Kingsborough Community College, LaGuardia Community College, 
and Queensborough Community College. These four colleges were chosen because each had op- 
erated the program for at least two years, had sufficient program infrastructure to continue oper- 
ating it, and was willing and able to participate in a random assignment evaluation. These are also 
the four largest CUNY community colleges by total enrollment. 


Borough of Manhattan Community College is located in lower Manhattan. It served 
about 29,000 students when the study started, including students in credit programs leading to a 
degree and Continuing Education programs. Kingsborough Community College is located in 
Brooklyn. It served about 25,000 students at the start of the study. LaGuardia and Queensborough 
Community Colleges are both located in Queens and served about 36,500 and 19,000 students, 
respectively, at the start of the study.** At Borough of Manhattan and Queensborough Community 
Colleges, most students take full-time course loads, whereas about half of Kingsborough and 
LaGuardia students do. All four colleges offer a wide range of associate’s degree programs that 
prepare students to transfer to four-year colleges or enter professional careers. As mentioned 
above, CUNY Start is housed in each college’s Continuing Education division. 


Research Questions and Design 


The evaluation will answer the following questions: 


e What is the effect of the option to participate in CUNY Start on students’ aca- 
demic outcomes, compared with the colleges’ standard courses and services 
(including the colleges’ standard developmental education courses)? Do the 
effects vary among student populations (for example, full-time and part-time 
students, or students with different degrees of developmental education re- 
quirements) and settings (individual colleges)? 


43See City University of New York (2018a). 


“City University of New York Office of Institutional Research and Assessment email to author, January 3, 
2018; City University of New York Office of Institutional Research and Assessment (2018). 
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e How is CUNY Start implemented? To what degree is it implemented with 
fidelity to the CUNY Start model? To what degree are there differences be- 
tween the program and the colleges’ standard courses and services, including 
the colleges’ standard developmental education courses? 


e What are the costs associated with CUNY Start? How do the costs compare 
with the costs of the standard courses and services, including the colleges’ 
standard developmental education courses? Is CUNY Start cost-effective? 


To measure the effects of CUNY Start relative to CUNY’s standard courses and services, 
the evaluation is using a random assignment research design. As is described below, eligible, 
interested individuals were randomly assigned either to a program group, whose members had 
the opportunity to participate in CUNY Start, or to a control group, whose members could not 
participate in CUNY Start but had the opportunity to participate in all of their colleges’ standard 
courses and services. The evaluation is comparing the academic outcomes of the program group 
and control group over time to determine the effect of CUNY Start. Because assignment to the 
two groups was random, the two groups were similar at the start of the study.*° Therefore, differ- 
ences between the average future outcomes of program group members and control group mem- 
bers yield an unbiased estimate of the effect of CUNY Start. Both groups of students are being 
tracked for at least two years to determine CUNY Start’s effects on persistence in college, college 
credit accumulation, and graduation. 


The Random Assignment Process and the Sample Members 


CUNY Start is available to incoming students whose scores on placement tests in math, 
reading, and writing show that they require remediation.*© CUNY Start’s full-time program, 
which covers math, reading, and writing, is only open to students who are assessed as needing 
remediation in math and at least one other subject area. CUNY Start’s part-time program, which 
covers math or reading and writing, is open to students who are assessed as needing remediation 
in at least one subject area. Students who are eligible for the full-time program can also choose to 
participate in the part-time program. Data from CUNY show that about two-thirds of the students 
who had participated in the part-time program through fall 2016 had had two or three develop- 
mental requirements.*’ Although any student who requires remediation is eligible for CUNY Start 


45s is described in Appendix A, the program group and control group had similar measured characteristics 
when they were randomly assigned. 


46Although CUNY Start targets first-time students, the program admits a small number of students with 
limited college experience (fewer than 12 credits) — for example, students who have taken and failed develop- 
mental courses in the past or students who are continuing their education after having stopped for some time. 


47City University of New York Office of the Senior University Dean for Academic Affairs (2017). 


14 


(and the evaluation), CUNY Start staff members make a priority of recruiting a narrower popu- 
lation of students: incoming students who have been assessed as needing remediation in all three 
subject areas (sometimes referred to as “triple remedial” students). 


For this evaluation, students were randomly assigned in three cohorts, before each of 
three consecutive semesters: spring 2015, fall 2015, and spring 2016.** Before each of these se- 
mesters, CUNY Start staff members at the colleges made contact with eligible students by mail 
and phone to describe the program and the evaluation. Interested students attended information 
sessions at the colleges to learn more. Students who agreed to participate in the study signed 
informed consent forms and participated in one-on-one interviews with CUNY Start staff mem- 
bers to confirm that they were eligible and did not have substantial barriers to participation, such 
as full-time jobs or extensive family care demands. Then students were randomly assigned to 
either the program group or the control group (using a secure internet link to a computer program 
at MDRC).“? Roughly four of every five eligible, consenting students were assigned to the pro- 
gram group and roughly one of every five was assigned to the control group. This division max- 
imized the number of students who could participate in CUNY Start while ensuring a large 
enough control group for analysis purposes. 


A total of 3,835 students are in the evaluation sample — 2,997 in the program group and 
838 in the control group. Table 1.1 shows the sample size by research group and college (with 
the college names masked).°° It also shows the proportion of sample members who said — just 
before random assignment — that if they were randomly assigned to the program group they 
intended to participate in the full-time CUNY Start program and the proportion who said they 
intended to participate in the part-time program. As the table shows, most sample members in- 
tended to participate in the full-time program. (As Chapter 3 discusses, about three-fourths of the 
program group students participated in their intended CUNY Start program.)* 


48Study enrollment began in spring 2015 and includes two spring semesters and one fall semester because 
of the timing of the grant. 


4°The research design allowed colleges to exempt some students from random assignment; 558 exempted 
students were admitted directly into CUNY Start and are not part of the research sample. Some students who 
were exempted from random assignment had previously participated in CUNY Start, some attempted to sign up 
for CUNY Start after sample enrollment had ended for a given semester, some were admitted on “discretionary” 
exemptions defined by each college, and some were previously in the CUNY Start part-time program and re- 
turning to complete the other subject area. 

>°The college names are masked in the tables in this report because it is not important to link outcomes with 
specific colleges, even though it can be useful to know whether impacts vary from college to college. 


51s Chapter 3 also discusses, the pre-random assignment intent to participate in the full-time or part-time 
CUNY Start program did not necessarily correspond to enrolling in college full time or part time among control 
group members. 
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Table 1.1 
The Evaluation Sample by College and Enrollment Type 


Program Control 


College Group Group Total 
All colleges 

Full-time program 1,841 514 2,355 

Part-time program 1,156 324 1,480 
College A 

Full-time program 519 146 8665 

Part-time program 391 111 502 
College B 

Full-time program 339 101 440 

Part-time program 0 0 0) 
College C 

Full-time program 283 82 365 

Part-time program 214 53 267 
College D 

Full-time program 700 185 885 

Part-time program 551 160 711 
Sample size 2,997 838 3,835 


SOURCE: MDRC random assignment database. 


NOTES: Of the total sample, 38 students withdrew from the study or did not 
have the correct consent forms, and are not included in analyses or tables in 
this report. Five hundred and fifty-eight students received "exemptions" and 
were offered the possibility of participating in CUNY Start without going through 
the random assignment process; these students are also excluded from 
analyses and tables. 

The college names are masked in the tables in this report because it is not 
important to link outcomes with specific colleges, even though it can be useful 
to know whether impacts vary from college to college. 


Students assigned to the program group were immediately provided a checklist of the 
steps they needed to complete to sign up for CUNY Start, which typically included registering 
for CUNY Start and making the $75 payment. Students assigned to the control group were di- 
rected to someone at the college who could provide information about the different courses and 
programs available and information about how to register. Control group students could take their 
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colleges’ regular developmental education and other courses and could participate in special pro- 
grams for which they were eligible. On a survey administered during sample members’ first se- 
mester in the study, most control group members (more than two-thirds) reported that they did 
not participate in any special programs.” 


Table 1.2 presents some baseline characteristics of the evaluation sample, drawn primar- 
ily from the CUNY Start application that students completed just before random assignment. 
Over half of the sample members are women and just under half were 19 years old or younger 
when they entered the study. The sample is racially diverse and a substantial proportion reported 
that their native language was not English. Over three-fourths of the sample members reported 
that they lived with their parents and few reported that they had any children. Just over one-third 
said they were the first in their family to attend college. Virtually all of the sample members 
reported that they intended to earn at least an associate’s degree, and most planned to earn a bach- 
elor’s degree or higher. Appendix Table A.1 presents more information on sample members’ 
baseline characteristics. 


Compared with all first-time students entering the four colleges in the study in the fall 
2015 semester, a higher proportion of students in the research sample are 19 or younger. This 
disparity probably reflects the fact that many in the research sample are recent high school grad- 
uates. Also, compared with all first-time students at the study colleges, a smaller proportion of 
students in the research sample identified themselves as white or Asian/Pacific Islander.** At 


5*Among control group members who said they were in special programs, the most common were ASAP, 
which is described in footnote 14 of this chapter; College Discovery, which provides academic support to stu- 
dents whose educational or financial circumstances might otherwise prevent them from attending college; the 
CUNY Language Immersion Program, in which English as a Second Language students receive English lan- 
guage instruction in context in specific content areas such as American history or environmental studies; learning 
communities, in which students enroll together in clusters of courses, and course readings and topics are linked 
by common themes; and immersion, in which students receive free intensive instruction to build their math, 
reading, or writing skills during the summer or between terms. All incoming students at Queensborough Com- 
munity College participate in one of five “Academies,” organized by major, that provide advising and learning 
activities such as collaborative projects and undergraduate research opportunities. Because all incoming students 
participate in Academies, they were not considered a “special program” on the student survey. 

°3Most sample members completed the CUNY Start application before random assignment. Fourteen per- 
cent of sample members, however, completed the application after random assignment. These sample members 
are included in the measures in Table 1.2 to provide descriptive information about the full research sample. 
Students who did not answer particular questions or who never completed applications are excluded from the 
calculations shown in Table 1.2. In contrast, Appendix Table A.1 includes all sample members in the calculations 
but excludes data drawn from applications that were completed after random assignment. As a result, the two 
tables present somewhat different numbers. 


>4Statistics for first-time students at CUNY’s community colleges in the fall 2015 semester are from the 
Integrated Postsecondary Education Data System. 
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Table 1.2 
Baseline Characteristics of the Evaluation Sample 


Percentage of the 


Characteristic Full Sample 
Female 56.2 
Age 
19 or younger 47.8 
20 to 23 30.1 
24 or older 22.1 


Race/ethnicity* 


Hispanic 43.7 
White 6.9 
Black 31.6 
Asian or Pacific Islander 10.2 
Other? 76 


Native language 


English 56.3 
Spanish 21.6 
Other 22.1 
Lives with parents 77.6 
Has one or more children under the age of 18 11.4 
Is currently employed 49.1 
Is the first person in the family to attend college 34.6 


Developmental subject areas required 


Zero (fully college-ready) 0.2 
One 15.2 
Two 33.4 
Three 51.2 


(continued) 
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Table 1.2 (continued) 


Percentage of the 


Characteristic Full Sample 


Highest level of education student hopes to achieve 


Some college (without degree) 0.5 
Associate's degree 13.5 
Bachelor's degree 59.0 
Postgraduate or professional degree 27.1 
Sample size 3,835 


SOURCES: CUNY Start application data, MDRC random assignment data, and test data from 
CUNY's Administrative Data Warehouse. 


NOTES: The overall sample size shown is for the full research sample, but different measures 
may have different sample sizes. Data on age and developmental subjects needed are available 
for the full sample. Missing data rates for other measures range from 14.5 percent to 23.0 percent. 
For full details see Appendix Table A.1. 

All values are weighted to account for random assignment ratios that vary across random 
assignment blocks. 

An omnibus F-test was conducted to see whether students’ baseline characteristics were jointly 
predictive of students’ random assignment status. The results were not statistically significant (p = 
0.885). 

aRespondents who selected Hispanic for their ethnicity and chose another race category are 
included only in the Hispanic category. Respondents who did not select Hispanic for their ethnicity 
and chose more than one racial category are included in the Other category. 

‘Other includes "multiracial," "Native American/Alaskan Native," and other racial/ethnic 
categories. 


CUNY, white and Asian students are approximately half as likely to be assigned to developmental 
education as are black and Hispanic students. 


Table 1.2 also shows in how many subject areas sample members required remediation 
when they entered the study. Based on their scores on placement tests, the vast majority of the 
research sample required remediation in multiple subject areas: 51 percent required remediation 
in math, reading, and writing, and 33 percent required remediation in two of the three subject 
areas. Reflecting CUNY Start’s target population, the proportion of sample members assessed as 
having multiple developmental education requirements is much higher than the proportion in the 
incoming student population at the four study colleges. Only 15 percent of first-time freshmen in 
fall 2015 at the four study colleges were assessed as needing remediation in math, reading, and 


55CUNY Task Force on Developmental Education (2016). 
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writing, and 21 percent were assessed as needing remediation in two subject areas (not shown in 
the table). 


Within the research sample, sample members who intended to participate in CUNY 
Start’s full-time program required remediation in more subject areas than those who intended to 
participate in the part-time program (not shown in the table). Among sample members who in- 
tended to participate in the full-time program, about two-thirds required remediation in three sub- 
ject areas and one-third required remediation in two subject areas. Among those who intended to 
participate in the part-time program, roughly one-third required remediation in three subject areas, 
one-third required remediation in two, and one-third required remediation in one. 


The research team also examined how far behind “college level” sample members were 
when they entered the study, as indicated by their scores on the placement tests. Sample members 
who were assessed as needing remediation in math had very low test scores; the vast majority had 
test scores far below the cutoff used to determine readiness for college-level math.5° Compared 
with the broader student population who required remediation at the four study colleges, sample 
members were about twice as likely to have very low math scores. In reading and writing, on the 
other hand, roughly half of the sample members who were assessed as needing remediation 
scored near those cutoffs.°” These proportions are similar to the proportions among the broader 
population of incoming students at the four colleges who required remediation in reading or 
writing. 


Data Sources 


The analyses in this report rely on multiple data sources. The data on sample members’ 
characteristics, described above, were collected using the CUNY Start application. The imple- 
mentation research discussed in Chapter 2, draws from a range of qualitative and quantitative data 
sources: interviews and focus groups, classroom observations, observations of CUNY Start pro- 
fessional development sessions, CUNY Start’s written descriptions of its curricula and other doc- 
uments, course syllabi, a student survey, and an instructor survey. The impact analysis, described 
in Chapter 3, relies on CUNY student transcript data and test data. See Appendix B for a descrip- 
tion of all the data sources. 


°6Only 9 percent of the sample members who were assessed as needing remediation in math scored within 
10 points of the college-ready score of 40. Fifty-four percent scored only 15 to 19, the lowest observed scores on 
the test. 


°7Forty-three percent of the sample members who were assessed as needing remediation in reading scored 
within 10 points of the cutoff, and 53 percent of the sample members assessed as needing remediation in writing 
scored within 8 points of the cutoff. 
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The Rest of This Report 


The remainder of this report presents findings from the evaluation and some preliminary 
conclusions. Chapter 2 discusses how the colleges in the study implemented CUNY Start and the 
extent to which they implemented the program with fidelity to the model. It also highlights 
important differences between the experiences of program group members and the experiences 
of control group members — the service contrast. Chapter 3 discusses the early impact findings, 
including CUNY Start’s effects on progress through developmental education and earning 
college-level credits in the first semester after random assignment, its effects on enrolling at a 
CUNY college in the second semester, and its effects among certain subgroups of students. 
Chapter 4 provides some conclusions based on the implementation and early impact findings. 
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Chapter 2 


Implementation Findings 


As described in Chapter 1, the CUNY Start model is intended to prepare students for college- 
level courses within a single semester, and it differs substantially from typical developmental 
education programs. Whereas typical developmental education courses run between 3 and 6 hours 
a week, full-time students attend CUNY Start for up to 26.5 hours a week — approximately 12 
hours for math, 12 for an integrated reading and writing class, and 1.25 to 1.5 for a college success 
seminar. In addition to its unique structure and time intensity, the CUNY Start model offers a 
different experience for both students and staff members with its distinct curricula and pedagog- 
ical practices, comprehensive student support, and robust staff recruitment and training. This 
chapter examines whether these elements of the CUNY Start model were delivered as intended 
during the study period and compares them with what students and staff members experienced in 
standard developmental course offerings and services. ! 


CUNY Start’s curricula stress conceptual understanding through activities that empha- 
size real-world learning, academic skill building, and mastery in the disciplinary content areas. 
Both reading/writing and math use instructional approaches that shift responsibility for teaching 
and learning from the instructor to the student, although exactly what that means in practice is a 
little different in reading/writing than it is in math. In addition to preparing students for college 
academically, CUNY Start aims to build their confidence and general college know-how through 
individual advising, a college success seminar, tutoring, and in-class academic assistance. The 
program’s staffing strategy allows CUNY Start to hire instructors and advisers who are open to 
the program’s distinct instructional and advising approach, and then gives them training and sup- 
port to help them implement that approach. In all these ways, CUNY Start seeks to build students’ 
foundational knowledge and conceptual understanding in math, reading, and writing and help 
them acquire the academic habits and confidence needed to persevere in college. 


The implementation research was designed to measure fidelity to the program design (or 
the extent to which CUNY Start’s offered services matched what was planned) and to measure 
service contrast (or the difference between the services experienced by the program group and 
control group) — both of which can potentially determine the program’s measurable effects. The 
implementation analysis draws on 134 interviews and focus groups with instructors, CUNY Cen- 
tral Office staff members, advisers, and students in both the CUNY Start (program) and non- 
CUNY Start (control) conditions. Complementing these data are 46 observations of CUNY Start 
classrooms and non-CUNY Start developmental math, reading, and writing classrooms; student 


‘The evaluation compares CUNY Start with other courses and services available at the colleges, including 
standard developmental education and several reformed courses at some of the colleges. 
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and instructor survey responses; and a review of curricular documents from CUNY Start and non- 
CUNY Start courses. (See Appendix C for a complete discussion of the implementation research 
methods.) 


Summary of Findings and Organization of This Chapter 


The implementation research found that the CUNY Start model was implemented as intended 
across the four colleges and that the experiences of CUNY Start students were different from 
those of non-CUNY Start students. Main findings include the following: 


e CUNY Start math instructors mostly followed the teaching methods they were 
expected to use. 


e The CUNY Start math curriculum’s focus on concepts distinguishes it from 
the usual developmental math courses, as does its focus on study skills. 


e Teaching methods in CUNY Start and non-CUNY Start math courses were 
starkly different. Non-CUNY Start instructors were more likely to rely on 
lecturing. 


e CUNY Start reading/writing instructors implemented the program’s teaching 
techniques as intended. 


e The integrated nature of the CUNY Start reading/writing curriculum distin- 
guishes the course from the usual developmental reading and writing courses, 
as does the use of writing exercises that position students as apprentices learn- 
ing to be proficient readers and writers. 


e In some instances, the instruction in non-CUNY Start reading and writing 
courses appeared similar to the instruction in CUNY Start reading/writing 
classes, particularly when it came to student-led discussions. 


e CUNY Start students experienced the multipronged student support model to 
varying degrees; however, non-CUNY Start students received less support 
overall. 


e The vast majority of CUNY Start instructors participated in CUNY Start’s 
training and continuing professional development, which proved distinct from 
what was offered to non-CUNY Start developmental faculty members. Most 
notably, non-CUNY Start instructors were far less likely to participate in an 
apprenticeship before teaching a course for the first time. 
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Table 2.1 summarizes the service contrast between CUNY Start and non-CUNY Start. 


The following sections describe the program’s instructional approaches (its curriculum 
and pedagogy) in math and reading/writing, the student support it offers, and its staffing and 
training components. They also describe whether these parts of the model were implemented as 
intended and the differences between CUNY Start and non-CUNY Start services. 


Instructional Approach to Math 


The CUNY Start math class teaches arithmetic and elementary algebra skills in one semester to 
help students prepare for college-level work more quickly than they could in a typical develop- 
mental sequence of classes. It aims to help students develop a deep understanding of mathematical 
concepts, and the academic skills they will need for college success. The curriculum advances 
toward these goals by using real-world examples and word problems that foreground fundamental 
mathematical concepts and numeracy skills without relying on formulas and memorization. To 
further build students’ academic skills, the class includes directions for taking notes and has in- 
structors review these notes. It relies primarily on “student-centered” instruction: Instructors pro- 
mote meaningful student discussion and engagement with the material, and frequently ask spe- 
cific, open-ended questions to stimulate student discussion. Implementation research found that 
CUNY Start instructors employed the curriculum and pedagogy as expected, and that the CUNY 
Start instructional approach differs substantially from non-CUNY Start developmental math 
course offerings. 


Curriculum 


The CUNY Start math curriculum consists of 47 prescribed lessons in Phase I that incor- 
porate arithmetic, functions, and elementary algebra. During this phase, instructors are expected 
to adhere to the curriculum strictly, teaching the same activities and materials in the same se- 
quence, with the same approaches to topics. The curriculum includes explicit guidance for in- 
structors, and worksheets and extra practice problems for students to complete in class and for 
homework. 


The curriculum often uses examples related to finance to illustrate concepts that can oth- 
erwise seem abstract. In one assignment designed to reinforce students’ understanding of positive 
and negative numbers, for example, instructors ask students to figure out how much change they 
would receive if they had $10 and owed a clerk $7. This assignment is one component of a more 
involved unit where financial examples are used to help students make use of their everyday un- 
derstanding of concepts that can otherwise seem abstract. In addition to providing problems like 
these designed to improve conceptual understanding, the curriculum also includes assignments 
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Table 2.1 


Summary of Service Contrast 


Component CUNY Start Non-CUNY Start 
College-level course Students cannot take college-level Students can take college-level 
enrollment courses while particpating in the courses while enrolled in 


Developmental math 
instructional 


approach 
Time in class 


Curriculum 


Pedagogy 


Developmental 


reading/writing 
instructional 


approach 
Time in class 


Curriculum 


Pedagogy 


Student support 
Success seminar 


Individual advising 


Academic resources 


program 


12 hours per week 


Integrated arithmetic and algebra; 
problems emphasize conceptual 
understanding; assignments include 
activities that develop students’ 
academic skills 


Mostly student-centered instruction 


12 hours per week 


Integrated reading/writing; nonfiction 
articles or essays and short stories; 
metacognitive logs and discussion 
journals 


Mostly student-centered instruction 


Mandatory; taught by adviser 


Student-to-adviser ratio 75:1; 
advising embedded in the program; 
focuses on academic and 
nonacademic topics and challenges; 
most surveyed students had at least 
one one-on-one advising session in 
the past semester 


Almost half of students received any 
tutoring; of those, 75 percent 


developmental education 


3-6 hours per week 


Arithmetic and algebra taught 
separately; academic skill-building 
activities not prevalent 


Some student-centered instruction; 
mostly lecture-based techniques 


3-6 hours per week 


Reading and writing content are not 
integrated at three of the four 
colleges; full-length works of fiction 
and textbooks on reading 
comprehension, grammar, or 
rhetoric; and upper-level 
developmental writing assignments 
include a research synthesis paper 


Mostly student-centered instruction 


Not mandatory for many students; 
taught by an instructor 


Student-to-adviser ratio 
approximately 600:1; advising 
offered by colleges; focuses on 
course registration; many surveyed 
students reported meeting with an 
adviser at least once in the past 
semester 


About a third of students received 
any tutoring; of those, 43 percent 


received math tutoring received math tutoring 


(continued) 
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Table 2.1 (continued) 


Staffing and training 


Staff recruitment Most instructors had master’s Most instructors had master’s 
degrees; 7 years of teaching degrees, but were also more likely 
experience, on average, 3 of which than CUNY Start instructors to have 
were at the college level doctorates; 20 years of teaching 


experience, on average, 13 of which 
were at the college level 


New instructor training 89 percent of surveyed instructors Most interviewed instructors did not 
participated in a semester-long participate in training before teaching 
apprenticeship a course for the first time 

Continuing instructor Surveyed instructors reported Surveyed instructors reported 

training participating in professional participating in professional 
development offerings during the development during the semester 


semester and received extensive 
comments on their instruction 


SOURCE: Community College Research Center field research data; MDRC calculations using data 
from the instructor and student surveys. 


meant to develop students’ study skills. Instead of using a textbook, students receive an empty 
binder during orientation, which they are required to fill with notes taken in class and course 
handouts. Instructors collect and review the binders before each test and comment on students’ 
organization and note taking, and on whether students corrected mistakes on assignments. 


Classroom observations and survey data show that CUNY Start math instructors imple- 
mented the curriculum as directed, teaching the content at the prescribed pace and using the as- 
signments and assessments provided. In interviews, instructors reacted positively to the curricu- 
lum and said they rarely deviated from it. When asked about their experiences with the 
curriculum, most CUNY Start instructors supported the curriculum’s focus on concepts and noted 
that they adhered closely to its materials. When they did deviate, it tended to be by moving 
through the material faster or slower than prescribed. Several instructors said it could be difficult 
to cover all the material in the allotted time. Others reported adjusting the rate at which they 
moved through some activities or topics depending on students’ understanding of the material. 


Survey responses indicate that most CUNY Start math instructors used real-world prob- 
lems. Eighty-three percent of instructors said they incorporated real-world examples when intro- 
ducing content, and 76 percent of program group student survey respondents who said they at- 
tended a math class reported that their instructor used such examples (see Tables 2.2 and 2.3).* 


See Box 2.1 for an explanation of how to read these tables. 
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Table 2.2 


Math Curriculum and Pedagogy 
Instructor Survey Results 


Sample CUNY Non-CUNY 


When introducing content, instructor often or always 
uses examples from the real world 119 83.3 71.9 11.4 


When asking students to practice or apply new content, 
instructor often or always 
Applies mathematical concepts to 
real-world problems 120 53.3 54.4 -1.1 


Asks students to explain out loud how they solved 
problems 120 96.7 64.4 32.2 


Asks students to discuss ideas and answers 
with fellow students 121 96.7 63.7 32.9 


Sample size (total = 186) 34 152 
SOURCE: MDRC calculations using data from the instructor survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Italic type indicates nonexperimental data. Significance tests are generally not calculated for 
nonexperimental data. 


Seventy-two percent of program group student survey respondents also reported applying math- 
ematical concepts to real-world problems when practicing or applying new concepts (see Table 
2.3). 


The curriculum differed substantially from that used in non-CUNY Start developmental math 
courses in several ways. First, non-CUNY Start courses were less likely to incorporate real-world 
problems; only 72 percent of instructors and 46 percent of students in non-CUNY Start courses 
reported using them. Second, the curriculum in non-CUNY Start developmental math courses 
was less standardized within and across colleges. Non-CUNY Start instructors typically followed 
a department-wide syllabus and used a recommended textbook; however, many included supple- 
mental or alternative materials and revised the curriculum from semester to semester based on 
students’ responses and performance. Third, non-CUNY Start math courses were more likely to 
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Table 2.3 


Math Curriculum and Pedagogy 
Student Survey Results 


Sample Program Control 


Measure (%) Size Group Group Difference P-Value 


Attended any math class 2,098 76.2 58.1 18.2 *** <0.001 


Among those who attended a math class 
When introducing content the instructor often or always 
Uses examples from the real world 1,485 76.0 52.8 23.2 


Encourages student-led discussion 1,483 87.1 64.2 22.9 


When asked to practice or apply new concepts, 
student often or always 
Applies mathematical concepts to real-world 
problems 1,481 71.6 46.4 25.2 


Explains his or her reasoning or thinking in 
solving a problem orally or in writing 1,485 84.1 60.2 23.9 


Works in groups of two or more to practice 
mathematical concepts 1,483 71.9 33.3 38.6 


Discusses mathematical concepts or 
procedures with other students in groups of 
two or more 1,484 73.3 33.9 39.4 


Sample size (total = 2,098) 1,480 618 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random assignment 
blocks and survey sample selection. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. See 
Appendix E for details on the impact-estimation model. 

Italic type indicates nonexperimental data. Significance tests are generally not calculated for 
nonexperimental data. 
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Box 2.1 


How to Read the Survey Tables in This Report 


Two types of surveys are discussed in this report — one that was given to students in the study 
and one that was given to instructors. The tables showing their findings appear in this chapter. 


The student survey was given to nearly all control group members and to a random sample of 
program group members. The student survey tables use the format excerpted below. The first 
row shows that 2,098 students responded to the question about attending math class. Because 
both the program and control groups had a high and nearly identical overall response rate, the 
student survey results that include all survey respondents can be treated as true experimental 
findings from a randomized controlled trial. Differences in responses between program and con- 
trol group members can be considered an estimate of the effect of the program. 


For example, as shown in the first row of the table excerpt below, the rates of attending a math 
class were different in the program and control groups by 18.2 percentage points, which means 
that CUNY Start caused an estimated 18.2 percentage points more students to attend a math 
class compared with the attendance rate had they not been offered CUNY Start. That difference 
is the estimated effect of CUNY Start. It is estimated rather than true because it is not knowable 
what would have happened in the absence of CUNY Start. The control group is an excellent 
proxy for what would have happened, but uncertainty remains. 


Estimated effects marked with one or more asterisks are statistically significant. The number of 
asterisks corresponds with the p-value, which indicates the likelihood that an estimated effect at 
least as large as the one observed would have occurred by chance, if the true effect were zero 
(that is, if there were no true effect). One asterisk corresponds to a probability of 10 percent or 
less; two asterisks, 5 percent or less; three asterisks, 1 percent or less. In other words, asterisks 
(and thus statistical significance) indicate that it is likely that CUNY Start had an effect (positive 
or negative) on that outcome. 


The question in the second row was answered by 1,485 survey respondents — a subset of re- 
spondents who said they attended a math class. The italics indicate that these findings are non- 
experimental. Although random assignment ensures that the entire program group and the entire 
control group were similar at the outset of the study, it is possible that the subset of math class 
attendees in the program group were systematically different from the subset of math class at- 
tendees in the control group. Consequently, differences in their outcomes may be caused by the 
program or they may be caused by differences in the types of students who responded to this 
question. Nonexperimental findings present and contrast math class attendees’ descriptions of 
what happened, but the differences they reveal cannot necessarily be attributed to the program. 


In this case, among program group survey respondents who attended a math class, 76.0 percent 
reported that their math instructors often or always used examples from the real world, compared 
with 52.8 percent among control group survey respondents, a 23.2 percentage point difference. 


(continued) 
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Box 2.1 (continued) 


Student Survey Responses, All Respondents and a Subset of 


Respondents 


Sample Program Control 
Measure (%) Size Group Group __ Difference P-value 
Attended any math class 2,098 76.2 58.1 18.2 *** < 0.001 


Among those who attended a 
math class 
When introducing content the 
instructor often or always: 
Uses examples from the real 
world 1,485 76.0 52.8 
Sample size (total = 2,098 1,480 618 


The instructor survey was given to all CUNY Start instructors and a random sample of non- 
CUNY Start developmental math, reading, and writing instructors. Unlike the student survey, 
the instructor survey had a low response rate among the non-CUNY Start instructors, which 
means that the non-CUNY Start instructor survey results must be interpreted cautiously. It is 
quite possible that non-CUNY Start instructor survey respondents do not represent all non- 
CUNY Start instructors. 


introduce arithmetic and algebraic concepts sequentially rather than integrating them. For exam- 
ple, in one non-CUNY Start developmental math class, the first 10 weeks of the semester were 
devoted to performing operations with fractions, decimals, and percentages — teaching arithme- 
tic as a discrete topic. At three of the four colleges, arithmetic was taught in a separate develop- 
mental math course. Finally, note taking and other academic skill-building activities were present 
but less prevalent in non-CUNY Start math curricula and performance requirements. 


Pedagogy 


CUNY Start’s teaching methods are designed to give students a deep understanding of 
math concepts through student-centered instruction and questioning. Student-centered instruction 
requires instructors to limit their use of lectures and instead promote student discussion. Students 
are positioned as active participants in their own learning, given time to think and struggle, and 
encouraged to speak and respond to each other. Questioning is one technique used to enact a 
student-centered pedagogy; it requires the instructors to ask specific open-ended questions and 
pose well-conceived problems to stimulate student discussion and thinking. The questioning tech- 
nique is a way to help students arrive at the correct answer while building their conceptual under- 
standing, and it also gives students the opportunity to respond to each other and identify misun- 
derstandings. This approach depends on the instructor’s ability to be nimble, because it relies on 
extemporaneous responses and varied questions. 
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In the four CUNY Start programs in the study, math instructors mostly used the teaching 
methods as expected. Instructor interviews and classroom observations revealed that many in- 
structors implemented student-centered instruction by asking students to discuss and explain their 
ideas and answers, both in response to the instructor’s questions and with their peers. According 
to one CUNY Start math instructor, “The CUNY Start approach is student-centered: The students 
are the ones doing all the talking. It is not lecture-based. Instead of demonstrating or telling them 
what to do, we ask questions to make them discover what they need to know.” Instructors were 
also observed asking questions that encouraged students to think about underlying math concepts 
and justify their answers. 


In a few instances, math instructors said they had run into trouble with CUNY Start’s 
pedagogy. In particular, instructors reported that it could be challenging to continue using the 
questioning technique when students reacted negatively to it — for example, when students 
seemed frustrated by the questions. Several instructors also noted that because the pedagogy relies 
on student participation, it is important to have classes of the right size. One instructor explained 
that when a class includes only a few students, it can be difficult to generate robust discussion; 
however, when a class is too large, discussion can become unmanageable. 


CUNY Start instructors appeared to use student-centered instruction and questioning 
more than non-CUNY Start instructors did. In interviews, most non-CUNY Start instructors did 
not mention employing student-centered instruction. Instructor surveys found the same thing (see 
Table 2.4): 97 percent of CUNY Start math instructors reported that they asked students to explain 
their thinking and to discuss ideas and answers with fellow students, compared with 64 percent 
of non-CUNY Start math instructors. Time series data collected during classroom observations 
also suggest that CUNY Start instructors spent more of their class time engaged in interactive 
discussion rather than lecturing.? In the classes observed, CUNY Start instructors spent 68 percent 
of class time leading discussions or facilitating interactive learning and 2 percent of class time 
lecturing. Non-CUNY Start instructors spent 18 percent of class time leading discussions or fa- 
cilitating interactive learning and 62 percent of class time lecturing. Table 2.4 summarizes the 
time series data. 


On the student survey, control group students also reported less use of student-centered 
techniques than program group students. Program group survey respondents were more likely to 
say that when they were asked to practice or apply new concepts, they often or always explained 
their thinking, discussed math concepts with other students in groups, and worked in pairs or 
groups to practice math concepts. A higher percentage of program group survey respondents also 
reported that classes often or always included student-led discussion (see Table 2.3). 


Time series data were collected using a structured protocol to record course format, pedagogical ap- 
proaches, curricular components, and student engagement at five-minute intervals. 
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Table 2.4 
Math Observation Time Series Pedagogical Breakdown 


CUNY  Non-CUNY 


Measure (%) Start Start 
Instructor initiates individual work 13.5 2.9 
Instructor initiates small-group work 0.0 0.0 
Instructor leads interactive discussion 49.7 18.3 
Instructor leads lecture 2.2 61.5 
Instructor provides support to individuals 9.4 11.5 
Instructor provides support to small group 0.3 1.0 
Instructor facilitates interactive learning 17.9 0.0 
Students present prepared work 0.0 4.8 
Other 6.9 0.0 
Samplesize(total=23)  s—i“‘(“‘(‘(‘(i<i‘;7C2R}?*‘i‘i;™;”!”.C(U OS T!!!CU SU 


SOURCE: Community College Research Center field research data. 


Instructional Approach to Reading/Writing 


Non-CUNY Start developmental education has historically offered separate reading and writing 
courses. CUNY Start integrates the subjects into one class that meets 12 hours per week to rein- 
force the relationship between the two and allow students to move more quickly through their 
developmental requirements. The reading/writing instructional approach, based on the cognitive 
apprenticeship model,‘ is designed to have students “apprentice” as proficient readers and writers; 
the instructor helps students tap into the ways of thinking and techniques of expert readers and 
writers. The curriculum engages students in reading and writing using culturally relevant texts, 
metacognitive logs, and discussion journals (described below). The pedagogy relies on modeling 
and coaching strategies to encourage students to employ the techniques and thought processes of 
proficient writers and readers in their work. Implementation research found that CUNY Start in- 
structors employed the reading/writing curriculum and pedagogy as expected, and that the CUNY 
Start curriculum is somewhat different from that of non-CUNY Start developmental education 
courses, while the pedagogy is relatively similar. 


4Collins, Brown, and Newman (1987). 
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Curriculum 


The CUNY Start curriculum integrates reading and writing practices through as many as 
34 lessons designed to prepare students for college-level literacy tasks.° The curriculum specifies 
a set of readings and accompanying materials designed to help students identify important ideas, 
analyze texts, paraphrase authors’ points of view, and respond with their thoughts. To engage 
students as readers, the curriculum uses a range of culturally relevant texts, including nonfiction 
articles and essays in the beginning of the semester, and literature, particularly short stories and 
fiction, later in the semester. The curriculum gives instructors some choice and flexibility regard- 
ing which texts to teach by including a packet of materials (readings, activities, and assignments) 
that can be used in place of the main lesson or in addition to it. 


To develop expert readers and writers, the curriculum relies on two types of writing ex- 
ercises: metacognitive logs and discussion journals. Metacognitive logs are used to help students 
process what they read and formalize the way they interpret and respond to texts. Metacognition 
is the practice of thinking about one’s thinking; by engaging in metacognition, students can iden- 
tify their thought processes and practices in service of becoming more effective readers and writ- 
ers. For instance, one metacognitive worksheet asked students to explain their reading thought 
processes and how those processes influenced their reading experiences. Discussion journals are 
also frequently used in CUNY Start reading/writing classes to give students practice reading, 
synthesizing, and responding to texts. The curriculum provides instructors with writing prompts 
for the discussion journals; these prompts typically ask students to elaborate on or support their 
answers at home or in class. For example, one prompt reads: “Describe the three main characters 
using adjectives. For each character, you must support your description with at least three pieces 
of evidence and analyze each piece of evidence (9 quotations total).” Instructors use the discus- 
sion journals as the basis for in-class discussion of the text. In addition to these writing activities, 
the curriculum includes opportunities for students to complete short analysis papers that compare 
multiple texts. 


CUNY Start instructors implemented the reading/writing curriculum as intended. On the 
student survey, most program group students said that they often or always read books, short 
stories, or articles in class; practiced finding the main idea and supporting details in a reading; and 


°The CUNY Start reading/writing curriculum provides a “scaffolded” approach to preparing students for 
college-level literacy tasks, meaning that students are coached as they practice and internalize new strategies, 
taking on more and more responsibility for those strategies as the semester progresses. During the study, CUNY 
Start implemented a curriculum called “Psychology and Literature” in addition to the regular content. Including 
this curriculum, the total number of CUNY Start reading/writing lessons ranges from 25 to 34. 
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drafted and revised essays in class (see Table 2.5).° Instructors reported during interviews that the 
curriculum engaged students in the reading and writing process through culturally relevant read- 
ings such as The Narrative of the Life of Frederick Douglass and The Immortal Life of Henrietta 
Lacks. When they diverged from the official CUNY Start curriculum, instructors said they did so 
intentionally, by choosing assignments to supplement the main instructional activities. 


The CUNY Start reading/writing curriculum was similar in some ways to non-CUNY 
Start developmental reading and writing course curricula, and was notably different in others. The 
CUNY Start and non-CUNY Start curricula had similar learning objectives (including learning 
to write academic essays and reading fiction and nonfiction texts) and both targeted particular 
competencies, such as annotation of class texts; vocabulary development; and summarizing, 
analyzing, and responding to texts. Unlike in CUNY Start, however, reading and writing are 
offered as separate courses in three of the four colleges’ non-CUNY Start developmental 
offerings. The types of texts and activities used to build students’ skills also differed. The 
curricular analysis shows that non-CUNY Start texts tended to be longer than CUNY Start texts, 
and non-CUNY Start classes used a wider range of texts, including New York Times articles; full- 
length works of contemporary and classic fiction; full-length works of nonfiction; short readings 
drawn from journalism, psychology, and sociology; and textbooks focused on reading 
comprehension, grammar, or rhetoric. Moreover, while CUNY Start explicitly teaches 
metacognition through metacognitive logs, comparable assignments were not commonly used in 
non-CUNY Start classes. Finally, the writing assignments in CUNY Start and non-CUNY Start 
classes were slightly different. Non-CUNY Start developmental English courses that served 
students deemed close to being college-ready required each student to write a research synthesis 
in addition to the reading response papers that both CUNY Start and non-CUNY Start classes 
required. The CUNY Start curriculum does not include research papers as part of its academic 
assignments but rather includes a final analysis paper that asks students to synthesize material 
from assigned texts. 


On the survey, students were asked if they were enrolled in any English class, which could have included 
CUNY Start reading/writing, a standard (non-CUNY Start) developmental reading or writing class, or college- 
level English. Student transcript data show that among the students who reported taking an English class, 7 
percent were enrolled in a college-level English class that was not a “corequisite” remediation course (that is, a 
course that combined developmental education and college-level material). Furthermore, according to transcript 
data, 13 percent of control group survey respondents who reported taking an English class were enrolled in 
college-level English classes (that were not corequisite remediation courses), compared with 5 percent of 
program group respondents who reported taking an English class. Student responses (and particularly control 
group student responses) to the reading/writing portions of the survey therefore represent a mix of different types 
of English remediation courses and programs and college-level English courses. 
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Table 2.5 


Reading/Writing Curriculum and Pedagogy 
Student Survey Results 


Measure (%) 


Attended any reading, writing, academic literacy, 
English, or English language arts class 


Among those who attended a reading/writing class 


Student often or always 
Reads books, short stories, or articles 
in class 


Practices finding the main idea and supporting 


details in reading 
Drafts essays in class 
Works on essay revisions in class 


Practices using information from readings in 


essays 


Student felt moderately or very comfortable 


Asking questions in class 


Responding to questions even when unsure 


of the answer 


Responding to other students' comments or 


questions in class 


Sample size (total = 2,098) 


Sample Program 


Size 


2,098 


1,574 


1,572 


1,569 


1,574 


1,574 


1,574 


1,573 


1,574 


Group 


72.6 


91.1 


93.6 
80.3 


78.7 


88.0 


83.6 


76.2 


79.2 


1,480 


SOURCE: MDRC calculations using data from the student survey. 


Control 


Group 


81.1 


81.1 


87.3 
62.8 


63.7 


73.7 


78.4 


75.0 


71.7 


618 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random assignment 


blocks and survey sample selection. 


Difference P-Value 


-8.5 *** <0.001 


9.9 


6.3 


17.5 


15.0 


14.3 


5.2 


1.2 


7.6 


Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. See 


Appendix E for details on the impact-estimation model. 


Italic type indicates nonexperimental data. Significance tests are generally not calculated for 


nonexperimental data. 


36 


Pedagogy 


Like CUNY Start math, CUNY Start’s reading/writing course uses student-centered 
teaching methods. It relies on techniques such as modeling and coaching, both of which position 
students as apprentices learning the habits and processes of expert readers and writers. Modeling 
requires teachers to think aloud to show students the processes used by proficient readers and 
writers, such as identifying evidence to support a claim or theme, identifying the author’s point 
of view, interpreting a text, or talking about how specific parts of a story are connected to others. 
This technique can be used with the whole class, with small groups, or with individual students. 
Coaching requires instructors to offer responses, questions, and examples that direct students back 
to a text or writing exercise to deepen their understanding. As is the case in math, reading/writing 
instructors are expected to ask questions that uncover students’ understanding or misunderstand- 
ing rather than giving them the correct response. 


CUNY Start reading/writing instructors implemented these pedagogical techniques as 
intended. They modeled features of the writing process for students using a variety of techniques, 
for example, demonstrating writing strategies by using a document reader to draft along with 
students.’ Instructors also followed the coaching technique. In interviews, most CUNY Start in- 
structors said that they encouraged students to refer to the text when they had a question or wanted 
to justify an answer. 


CUNY Start and non-CUNY Start reading/writing classes had similar pedagogical ap- 
proaches. Many non-CUNY Start instructors described teaching practices similar to those used 
by CUNY Start instructors, such as having students work in small groups to discuss texts. Instruc- 
tor survey responses show that about the same proportions of non-CUNY Start instructors and 
CUNY Start instructors led interactive discussions (see Table 2.6). Program and control group 
survey respondents reported similar levels of comfort asking and responding to questions in class, 
and responding to other students’ questions and comments (see Table 2.5). Classroom observa- 
tion data suggest slight differences in the use of class time, however. They show that CUNY Start 
instructors spent more time leading discussion, initiating individual work, or providing support to 
individuals. It is more difficult to draw generalizations from observations of non-CUNY Start 
classes, since most were reading only and writing only (see Table 2.7).° Across this mix of non- 
CUNY Start classes, instructors lectured more often and, for the most part, initiated individual 
work and provided support to individuals less frequently. 


7A document reader is an electronic device that allows instructors to write notes on a document while pro- 
jecting it before the class. 

8One of the non-CUNY Start classes that was observed combined reading and writing. It is included as a 
data source for this chapter. However, no time-series data were collected during this observation, so the class is 
not included in Table 2.7. 
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Table 2.6 


Reading/Writing Pedagogy 
Instructor Survey Results 


Sample CUNY 


Measure (%) Size — Start 


When introducing content, instructor often or always 


Asks students to work collaboratively on activities 


or assignments during class 96 92.6 
Facilitates interactive discussion 97 96.3 
Sample size (total = 156) 32 


SOURCE: MDRC calculations using data from the instructor survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 


Non-CUNY 
Start Difference 
72.5 20.1 
97.1 -0.8 

124 


Italic type indicates nonexperimental data. Significance tests are generally not calculated for 


nonexperimental data. 


Table 2.7 


Reading/Writing Observation Time Series Pedagogical Breakdown 


CUNY Start 

Reading/ 
Measure (%) Writing 
Instructor initiates individual work 18.3 
Instructor initiates small-group work 7.6 
Instructor leads interactive discussion 35.7 
Instructor leads lecture 3.8 
Instructor provides support to individuals 14.8 
Instructor provides support to small groups 6.5 
Instructor facilitates interactive learning 5.7 
Students present prepared work 1.9 
Other 5.7 
Sample size (total = 21) 12 


SOURCE: Community College Research Center field research data. 
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Reading 


7.3 
7.3 
14.6 
19.5 
2.4 
0.0 
48.8 
0.0 
0.0 


3 


Non-CUNY Start 


Writing 


8.7 
13.0 
31.3 
23.5 

2.6 
17.4 

0.0 

0.0 

3.5 


6 


Student Support 


CUNY Start offers three main types of student support: a college success seminar, individual 
advising, and tutoring. To help them develop peer relationships, CUNY Start students also take 
their academic classes in a cohort that is supported by an assigned adviser who leads their weekly 
college success seminar. These components of the CUNY Start student support model were im- 
plemented as intended and were distinct from the forms of support available to non-CUNY Start 
students. 


College Success Seminar 


All CUNY Start students are required take a college success seminar with their cohort, 
led by their assigned adviser, for 1.25 to 2.5 hours per week. The seminar aims to acquaint stu- 
dents with college resources and help them develop the skills and dispositions to balance school 
and life commitments, solve problems, advocate for themselves, and view themselves as learners. 
Classroom observation data indicate that across colleges, CUNY Start seminar classes addressed 
similar concepts, such as researching and selecting a major and applying for financial aid. 


Program group survey respondents were more likely than control group respondents to 
say they had participated in a college success seminar (78 percent compared with 29 percent), 
and they rated the quality of the seminar higher than did control group survey respondents who 
took such a seminar, as shown in Table 2.8. In general, non-CUNY Start students were not re- 
quired to take college success seminars. Some non-CUNY Start students had access to college 
success seminars that covered content similar to CUNY Start’s, but these seminars were not usu- 
ally led by students’ advisers. At one college, students could participate in a first-year “learning 
academy” with other students in their year that introduced them to the college’s academic and 
nonacademic resources. At another college, students could participate in a seminar tailored to 
broad disciplines in the college (for example, health sciences, business, or liberal arts). 


Individual Advising 


In addition to leading the seminar class, CUNY Start advisers meet with students in the 
program individually. Individual advising creates an additional opportunity to help students ma- 
triculate and reach their academic and social goals. Advisers build relationships with students and 
their instructors that enable them to intervene if students are struggling. Program directors ex- 
plained that while they preferred that advisers meet with students at least twice a semester, there 
was no formal policy dictating how many times advisers must meet with students. Advisers meet 
weekly with the reading/writing and math instructors for their cohorts to discuss students’ pro- 
gress and plan for intervention if necessary. In individual sessions, CUNY Start advisers help 
students develop self-awareness and inform them about college resources such as financial aid, 
course registration, and college policies to aid their integration into the college once they complete 
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Table 2.8 


Student Seminar, Advising, and Tutoring Experience 
Student Survey Results 


Sample Program 


Measure Size 


Seminar 

Currently taking a weekly college seminar taught by the 
student's adviser, college success seminar, freshman 
seminar, or student development seminar (%) 2,065 


Among those who took a seminar 
Averaged 4 or higher out of 5 on the quality-of-seminar 
scale* (%) 1,306 


Advising 
Had at least one one-on-one advising session with 
an adviser (in person or on the telephone) (%) 2,045 


Median number of one-on-one advising sessions 2,045 


Among those who attended a one-on-one advising session 


Median amount of time spent talking with the 
adviser (minutes) 1,510 


Saw or spoke with the same adviser each 
time (%) 1,569 


Discussed the following topics” (%) 


Academic goals 1,616 
Academic progress 1,616 
Class attendance or punctuality 1,616 
Study skills or time management 1,616 
Course selection 1,616 
Major 1,616 
Requirements for graduation 1,616 
Internships 1,616 
Job opportunities 1,616 
Career planning 1,616 


College policies such as transfer credit 


policies, probation, and drop/add policies 1,616 
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Group 


78.2 


80.3 


82.1 


2.0 


15.0 


82.0 


74.6 
67.1 
55.5 
53.4 
45.9 
68.0 
40.9 
24.6 
31.6 
44.0 


32.4 


Control 


Group 


28.7 


64.8 


65.3 


1.0 


15.0 


40.7 


59.2 
45.3 
29.1 
33.8 
62.7 
73.4 
35.6 
14.6 
18.0 
25.7 


24.8 


Difference P-Value 


49.5 *** <0.001 


15.6 


16.8 *** <0.001 


1.0 *** <0.001 


0.0 


41.3 


15.4 
21.8 
26.4 
19.6 
-16.8 
-5.4 
5.4 
10.0 
13.6 
18.3 


7.6 


(continued) 


Table 2.8 (continued) 


Sample Program Control 


Measure Size Group Group Difference P-Value 


College services such as financial aid, 


tutoring, and counseling 1,616 48.6 34.8 13.8 
Personal matters 1,616 29.7 14.5 15.1 
Something else 1,614 4.4 3.9 0.5 
Averaged 4 or higher out of 5 on the 
quality-of-advising scale* (%) 2,027 67.3 39.4 27.9 
Tutoring 
Received any tutoring (%) 2,072 45.0 33.6 11.4 *** <0.001 


Among those who received tutoring 


Received tutoring in (%): 


Math 860 74.9 43.1 31.8 
Reading 860 44.4 37.6 6.7 
Writing 860 53.7 56.4 -2.7 
Some other subject 859 2.7 11.6 -8.9 
Median number of times student met with a tutor 818 3.0 4.0 -1.0 


Rated the tutoring received during the semester 
as moderately or very helpful (%) 861 87.6 74.9 12.7 


Sample size (total = 2,098) 1,480 618 
SOURCE: MDRC calculations using data from the student survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random assignment 
blocks and survey sample selection. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. See 
Appendix E for details on the impact-estimation model. 

Italic type indicates nonexperimental data. Significance tests are generally not calculated for 
nonexperimental data. 

aSee Appendix D for details on the survey scales. 

‘Distributions do not add to 100 percent because categories are not mutually exclusive. 
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the program. During the study, CUNY Start advisers managed caseloads of up to 75 students, 
compared with non-CUNY Start advisers, who could be responsible for several hundred students. 


The implementation of CUNY Start’s individual advising varied across colleges and 
sometimes within colleges. Absent a formal policy about how often advising should take place, 
CUNY Start advisers and administrators noted that they provided assistance more frequently to 
students who were facing challenges. An adviser explained: 


Usually, we have to see people when they have attendance issues, and we try to 
problem solve with them and figure out why it’s happening and what we can do 
to improve the situation. Meetings [can be] about discipline issues in the class- 
room. We try not to pull them out of class because we don’t want them to fall 
behind. 


Responses on the student survey corroborate these accounts (see Table 2.8). Sixty-seven 
percent of program group survey respondents who met with an adviser at least once reported 
discussing academic progress with that adviser, and 56 percent reported discussing class attend- 
ance or punctuality issues. 


CUNY Start’s individual advising model and smaller caseloads stood in contrast to non- 
CUNY Start advising services. A CUNY Start student has a sustained relationship with the same 
adviser throughout the semester, and this adviser is more likely than a non-CUNY Start adviser 
to know about the student’s academic progress and challenges, probably because non-CUNY 
Start advisers work with many more students. During the study, non-CUNY Start advisers indi- 
cated that while most colleges require incoming freshmen to meet with advisers to discuss their 
placement test scores and arrange their schedules, students were not required to attend additional 
individual advising sessions except for enrollment and course registration. 


Student survey responses offer additional information on the differences between CUNY 
Start and non-CUNY Start individual advising (see Table 2.8): 82 percent of program group sur- 
vey respondents reported having at least one individual advising session, compared with 65 per- 
cent of control group respondents. The median control group survey respondent had one individ- 
ual advising session during a semester, while the median program group respondent had two. 
CUNY Start students were also more likely to see or speak with the same adviser each time than 
control group students. Of those who reported seeing an adviser at least once, program group 
survey respondents were more likely to report having discussed academic goals and career plan- 
ning, while control group respondents more often reported discussing course selection. Students 
were asked to describe the quality of advising using several survey questions that were then com- 
bined to create a scale (see Appendix D). On average, 67 percent of program group survey re- 
spondents rated their advising highly on the scale, compared with 39 percent of control group 
survey respondents. In general, while some non-CUNY Start students had access to advising sim- 
ilar to that available to CUNY Start students, they needed to take more initiative to seek it out and 
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to develop a personal relationship with an individual adviser in which they could discuss nonac- 
ademic and personal issues. 


Academic Support 


In addition to the multifaceted support provided by advisers, CUNY Start offers students 
integrated academic resources. Writing assistants worked with the instructor in most observed 
reading/writing classrooms, providing individual support to students during class, grading assign- 
ments, and leading after-class tutoring sessions. Math tutors, who are former CUNY Start stu- 
dents, also circulated during observed classes to assist students, and administered after-class tu- 
toring. These support staff members operate as backup instructors when the lead instructor is busy 
or if they identify a struggling student. 


Overall, 45 percent of program group survey respondents reported receiving academic 
tutoring, compared with 34 percent of control group respondents (see Table 2.8). Among those 
who received tutoring, program group survey respondents were more likely than control group 
respondents to say they had engaged in math tutoring (75 percent compared with 43 percent). 
Eighty-eight percent of program group survey respondents who received tutoring rated that tutor- 
ing as moderately or very helpful, compared with 75 percent of control group respondents. 
Among those who received tutoring, the median number of meetings was 3 for program group 
survey respondents and 4 for control group respondents. While these CUNY Start and non- 
CUNY Start survey results appear similar, the integrated nature of CUNY Start’s academic sup- 
port services may mean that CUNY Start students receive tutoring more often. CUNY Start stu- 
dents may not identify every interaction in class with an embedded classroom writing assistant 
and or math tutors as “tutoring,” and as a result their survey responses may underrepresent their 
engagement with tutoring. Alternatively, CUNY Start students may not need as much tutoring as 
non-CUNY Start students because they are receiving more intensive developmental education 
instruction. 


Staffing, Training, and Professional Development 


CUNY Start instructors are expected to implement the curriculum as designed and use student- 
centered teaching practices to impart the thought processes of proficient mathematicians and sci- 
entists or readers and writers. CUNY Start advisers provide support through individual advising 
and by teaching the college success seminar. To meet these unique teaching and advising expec- 
tations, the program recruits instructors and advisers based on their pedagogical experience and 
their openness to the prescribed pedagogy, and provides them extensive training and continuing 
professional development. Professional development staff members from CUNY’s Office of Ac- 
ademic Affairs (hereafter called “professional developers”) work with campus programs to over- 
see the recruitment, hiring, and training of CUNY Start instructors and advisers. Staffing, training, 


43 


and professional development practices were implemented as expected and were distinct from 
what colleges’ full- and part-time developmental education faculty members and advisers expe- 
rienced. 


Staff Recruitment 


Staff members are recruited by the CUNY Central Office and campus programs working 
in concert. Professional developers participate in the screening and interview process, and 
campus-based CUNY Start directors and program coordinators make the final hiring decisions. 
Applicants must be prepared to teach reading, writing, or math and must be willing to implement 
CUNY Start’s curriculum and student-centered pedagogy. Applicants must also be interested in 
working with the CUNY Start population. According to one CUNY Start director, the primary 
reason potential instructors are turned away is that they prove unwilling to adjust to CUNY Start’s 
pedagogy. Professional developers report that instructors with extensive teaching experience are 
less willing to adopt a wholly new way of teaching, which could explain why CUNY Start 
instructors had less teaching experience on average (8 years total, 3 at the college level) than did 
non-CUNY Start instructors (15 years total, 12 at the college level). CUNY Start instructors 
tended to have teaching experience in both K-12 and higher education. Some had taught basic 
skills as adjunct professors or in adult education programs. Similarly, CUNY Start advisers often 
had experience working with adult learner populations through social work or had been high 
school guidance counselors. 


CUNY Start and non-CUNY Start instructors had different educational credentials as 
well. The majority of both CUNY Start and non-CUNY Start instructors had master’s degrees, 
but CUNY Start instructors were less likely to have doctorate degrees than non-CUNY instruc- 
tors. CUNY Start instructors were also on average 20 years younger than non-CUNY Start in- 
structors. Table 2.9 provides additional information about the characteristics of CUNY Start and 
non-CUNY Start instructors.° 


New Instructor Training 


Once hired, CUNY Start instructors are expected to participate in a one-semester inten- 
sive apprenticeship designed to provide them with the skills and understanding they need to im- 
plement the program’s pedagogy and curriculum effectively. This apprenticeship is a full-time 
paid position in which new hires learn new teaching strategies and become familiar with the cur- 
riculum. 


°Comparable data were not collected for CUNY Start and non-CUNY Start advisers. 
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Table 2.9 


CUNY Start and Non-CUNY Start 
Developmental Education Instructor Characteristics 


Sample CUNY Non-CUNY 
Characteristic Size Start Start 
Highest degree completed (%) 
Doctorate 212 3.7 28.5 
Master’s® 212 72.2 64.6 
Bachelor's 212 24.1 6.3 
No degree 212 0.0 0.6 
Female” (%) 211 69.8 54.4 
Median total years taught at any level 211 7 20 
Median total years taught at the college or graduate level 210 3 13 
Median age (years) 196 32 52 
Sample size (total = 342) 66 276 


SOURCE: MDRC calculations using data from the instructor survey. 


NOTES: 2A small proportion of respondents selected a professional degree (such as MD, DO, JD, 
etc.). Due to the small sample, these are included in the master's category. 

5A small proportion of respondents indicated a nonbinary gender identity; due to small sample 
issues these responses have been excluded from the proportions shown. 


During the early stages of the apprenticeship, newly hired instructors, referred to as co- 
operating instructors or “co-ops,” observe the lead instructor’s teaching daily. They discuss the 
lessons and specific practices with the lead instructor and at CUNY Central Office training ses- 
sions with other co-ops. As co-ops become comfortable with the curriculum and pedagogy, they 
gradually assume responsibility for teaching while the lead instructor observes, helping them gain 
independence and mastery. 


Reports from CUNY Start instructors indicate that this staged element of training took 
place as intended. One math instructor described initially spending time watching and listening 
and then, as the semester progressed, leading some of the class activities. Toward the end of the 
semester, he led an entire class. Similarly, when asked to describe her apprenticeship, a CUNY 
Start reading/writing instructor explained that in the beginning of the semester, she reviewed and 
discussed lesson plans with the lead instructor, and then eventually she managed several lessons 
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independently, after which she would receive comments on her performance from the lead in- 
structor. 


As aco-op begins teaching, both the lead instructor and CUNY Start professional devel- 
opers provide comments on the strengths and weaknesses of his or her teaching. These comments 
are designed to prepare co-ops to become instructors in the following term. If new instructors do 
not show proficiency with the pedagogy by the end of the co-op semester, they may be invited to 
stay on to support other teachers and continue to immerse themselves in the model until they are 
deemed ready to lead their own classes. 


The CUNY Start instructor-training model was generally implemented as planned. In- 
structor survey data indicate that all math instructors went through an apprenticeship period, as 
did 77 percent of reading/writing instructors (not shown in a table). According to CUNY Start 
administrators, some reading/writing instructors may have bypassed the co-op phase to accom- 
modate the rapid pace at which the program was expanding before the study began. In addition, 
CUNY Start allowed writing assistants to transition to teaching without apprenticing; however, 
these were rare exceptions to the training model and the practice was discontinued during the 
study. A former writing assistant explained that, because she had been with the program for sev- 
eral years, and because the program needed instructors, she was able to become a core instructor 
without apprenticeship training. In lieu of that training, she received extra support and frequent 
observations and comments on her performance from the professional developers. 


The CUNY Start training model is distinct from what is offered to non-CUNY Start de- 
velopmental education instructors. Interview data suggest that non-CUNY Start instructors were 
less likely to participate in training before teaching a course for the first time, apart from those at 
one college that mandated a yearlong seminar for new faculty members during their first year of 
teaching. At this college, new faculty members met once or twice a month to discuss curriculum 
planning and student learning. 


Instructor Professional Development 


CUNY Start provides continuing professional development through classroom observa- 
tions by professional developers and all-staff development events. Additionally, CUNY Start in- 
structors and advisers can participate in email listservs on which professional information can be 
shared. 


CUNY Start professional developers are expected to observe all instructors twice during 
their first semester of teaching and at least once in subsequent semesters. Professional developers 
noted that the purpose of the observations is to ensure that elements of the pedagogical model 
such as student-centered instruction are present, and to identify areas where instructors need to 
improve their teaching practice. A math professional developer explained that they want to see 
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instructors asking good questions and not giving students the answer too quickly. After the class- 
room observations, professional developers send their notes to the instructors and shortly there- 
after meet with them to discuss what worked well and what could be improved. 


In addition to undergoing classroom observations, CUNY Start instructors are expected 
to participate in at least four mandatory and several voluntary professional development oppor- 
tunities sponsored annually by the CUNY Central Office. These workshops address broad topics 
that apply across programs, such as training in interpersonal interactions and emotional intelli- 
gence to improve classroom management. Instructors are also encouraged to use email listservs 
to keep in touch and stay up-to-date throughout the year. 


Many CUNY Start instructors participated in this mix of professional development op- 
portunities, and a majority reported on the instructor survey that they participated in workshops 
that addressed broad topics. Almost all math instructors and 100 percent of reading/writing in- 
structors reported receiving comments on their instruction from administrators and colleagues 
(not shown in table). However, during interviews, several instructors raised questions about the 
consistency and purpose of these comments. Several instructors noted that the comments came 
more sporadically than anticipated, and that it was not clear whether the observations were in- 
tended to inform their teaching or evaluate their performance. 


Overall, CUNY Start and non-CUNY Start instructors had different opportunities for 
professional development. CUNY Start instructors were more likely than non-CUNY Start in- 
structors to receive comments on their instruction from administrators and colleagues (see Table 
2.10). Non-CUNY Start faculty members reported participating in fewer hours of professional 
development during a semester and described obstacles to participation, such as limited time due 
to teaching responsibilities. While non-CUNY Start instructors reported participating in profes- 
sional development activities similar to the workshops for CUNY Start instructors, these activities 
focused on different topics. Compared with CUNY Start instructors, non-CUNY Start instructors 
were more likely to say they had attended professional development workshops on designing 
effective lessons and assessments. See Table 2.10 for survey findings on instructors’ professional 
development. 


Adviser Training and Professional Development 


Professional development for CUNY Start advisers is focused on helping them 
implement the seminar effectively and conduct individual advising. New advisers participate in 
a semester-long apprenticeship, in which they learn to lead the seminar and gain experience in 
individual advising while gradually assuming more responsibility over these duties. While 
training for the seminar, some advisers take on a caseload of 50 students to learn the necessary 
advising responsibilities. Advisers in training are observed by senior advisers and professional 
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Table 2.10 


Professional Development Among CUNY Start and 
Non-CUNY Start Instructors 


Sample CUNY Non-CUNY 

Measure Size Start Start 
Went through an apprenticeship period as a cooperating instructor (%) 56 89.3 - 
Participated in workshops, institutes, or conferences for faculty members/ 
instructors at instructors’ college only (%) 217 73.7 70.6 
Attended workshops, institutes, or conferences organized outside 
CUNY (%) 216 28.1 46.5 
Participated in departmental meetings to discuss curriculum, 
instruction, or pedagogy (%) 217 86.0 66.9 
Received comments on instruction from administrators and 
colleagues (%) 215 98.2 67.1 
Attended professional development sessions on (%) 

Designing effective lessons 206 60.7 76.0 

Designing effective assessments 203 33.9 74.8 
Median total hours spent on professional development activities 202 16 10 
Sample size (total = 342) 66 276 


SOURCE: MDRC calculations using data from the instructor survey. 


developers to determine their readiness to handle their own caseload. During this time, CUNY 
Start advisers must also familiarize themselves with campus resources so that they can inform 
students about the range of services available. They also attend training sessions at the CUNY 
Central Office to supplement their on-campus training. 


Because CUNY Start had recently expanded to all the CUNY campuses, some advisers 
had to lead seminars and advise students during their apprenticeship semesters rather than only 
observing, working with, and learning from a lead adviser. Some of these advisers indicated that 
they would have benefited from more assistance during training. Due to limited data on the train- 
ing experiences of non-CUNY Start advisers, it is unclear what training opportunities were avail- 
able to those advisers and what professional development activities they participated in. 
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Chapter 3 


Short-Term Effects on Educational Outcomes 


CUNY Start aims to enable students with substantial developmental education requirements to 
dramatically reduce or eliminate those requirements in a single semester, preparing them for ac- 
ademic success in college-level courses. Students are asked to focus solely on progress through 
their developmental education requirements before working to earn college-level credits. After 
one semester, students participating in CUNY Start are expected to have made substantial pro- 
gress through those developmental education requirements in lieu of accumulating the college- 
level credits they may have earned otherwise. 


This chapter presents the effects of CUNY Start on students’ short-term academic 
progress. The primary measures of academic progress analyzed are enrollment, progress through 
developmental education, and college-level credit accumulation. At the time this report was 
written, data were only available for the program semester (when most program group students 
participate in CUNY Start) and the start of the first postprogram semester. (Box 3.1 provides 
additional details on how program and postprogram semesters are determined for students 
participating in CUNY Start or enrolled in non-CUNY Start courses.) Therefore, the analyses 
presented here only provide preliminary information on whether CUNY Start is producing the 
expected results. Follow-up data must be collected over a longer time to assess whether CUNY 
Start’s short-term trade-off ultimately improves students’ long-term educational progress and 
success. 


Summary of Findings 


These early analyses find the following: 


e CUNY Start is performing in a manner consistent with its theory of change: 
During the program semester, program group students made substantially 
more progress through their developmental education requirements than con- 
trol group students, while control group students accumulated more college- 
level credits. 


e During the first postprogram semester, program group students enrolled at 
CUNY colleges (that is, either participated in CUNY Start as nonmatriculated 
students or enrolled in non-CUNY Start courses as matriculated students) at a 
higher rate than control group students. This positive effect largely reflects the 
fact that program group students participated in CUNY Start at a higher rate 
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Box 3.1 


The Timing of Program and Postprogram Semesters 


CUNY Start follows the same structure at all the colleges in the study. As described in Chapter 
1, the program has two phases: Phase I involves 12 weeks of instruction. Students who do not 
pass tests at the end of Phase I receive an additional 3 to 6 weeks of instruction in Phase II. 


Students who have enrolled at a CUNY college take courses during terms defined by each col- 
lege. At Kingsborough Community College and LaGuardia Community College, the academic 
calendar is divided into two terms (fall and spring), each of which has a “main session” (gener- 
ally 12 weeks) and an “intersession” (usually 6 weeks). At Borough of Manhattan Community 
College and Queensborough Community College, the academic calendar is divided into four 
terms (fall, winter, spring, and summer), with fall and spring being longer terms (generally 14 
weeks) and winter and summer being shorter terms (from 4 to 8 weeks). Generally, at the end 
of each term students enrolled in developmental education courses at any CUNY college take 
exit tests for the subject areas in which they received remediation. 


In order to create a uniform set of metrics for students participating in CUNY Start and students 
enrolling in non-CUNY Start courses at CUNY colleges (as matriculated students), “semesters” 
are defined in the following way in the analyses in this report: For students participating in 
CUNY Start, the semester refers to course taking and performance on exit tests during Phase I 
and Phase II. For students enrolled in non-CUNY Start courses at CUNY colleges, at Kings- 
borough and LaGuardia the semester refers to course taking and performance on exit tests during 
both the main session and the intersession, and at Borough of Manhattan and Queensborough it 
refers to both the longer and shorter follow-up terms (fall and winter, or spring and summer). At 
all four colleges, CUNY Start generally started within two weeks of the college’s other courses, 
so defining semesters in this way results in mostly overlapping time periods. 


The study includes three cohorts of students: those who started in spring 2015, fall 2015, and 
spring 2016. These cohorts are also aligned in the analyses so that the program semester refers 
to the first semester after random assignment and the postprogram semester refers to the subse- 
quent term. The table below shows the program semester and postprogram semester for each 
cohort. 


Program Semester Postprogram Semester 
CUNY Start CUNY Start CUNY Start CUNY Start 
Cohort Phase | Phase Il Phase | Phase Il 
Spring 2015 
Kingsborough Spring 2015 Spring 2015 Fall 2015 Fall 2015 
and LaGuardia main session intersession main session intersession 


Queensborough Spring 2015 Summer 2015 Fall 2015 Winter 2016 
and Borough of 
Manhattan 


(continued) 
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Box 3.1 (continued) 


Fall 2015 
Kingsborough Fall 2015 Fall 2015 Spring 2016 Spring 2016 
and LaGuardia main session intersession main session intersession 


Queensborough Fall 2015 Winter 2016 Spring 2016 Summer 2016 


and Borough of 
Manhattan 


Spring 2016 
Kingsborough Spring 2016 Spring 2016 Fall 2016 Fall 2016 


and LaGuardia main session intersession main session intersession 
Queensborough Spring 2016 Summer 2016 Fall 2016 Winter 2017 
and Borough of 

Manhattan 


NOTE: Enrollment data are not available for the fall 2016 intersession or the winter session 
of 2017, but test data from this period are available. 


e thancontrol group students in that postprogram semester. The two groups en- 
rolled in non-CUNY Start courses as matriculated students at similar rates. 


e Exploratory analyses suggest that program group students learned more in 
math, reading, and writing than control group students by the end of the pro- 
gram semester. 


e Preliminary analyses suggest that certain types of students may have more to 
gain than others from the trade-off of making a priority of developmental ed- 
ucation before working on college-level credits. The trade-off seems to favor 
students planning to participate in the CUNY Start full-time program and stu- 
dents assessed as needing remediation in all three subject areas (math, reading, 
and writing). 


Analyses 


As discussed in Chapter 1, this study is a randomized controlled trial. Individuals were randomly 
assigned to one of two groups through a process that functions like a lottery. The program group 
was given the opportunity to participate in CUNY Start. The control group was given the oppor- 
tunity to take colleges’ usual courses and services outside of CUNY Start. Because of random 
assignment, the two groups of students were similar at the start of the study. Appendix Table A.1 
compares the characteristics of students in the program and control groups, as reported by students 
on the CUNY Start application before random assignment. As shown in the table and discussed 
in more detail in Appendix A, most of the program and control groups’ measured characteristics 
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were very similar. As a result of random assignment, differences between the average future out- 
comes of all program group members and all control group members constitute an unbiased es- 
timate of the effect of the opportunity to participate in CUNY Start. 


This chapter addresses two main topics. Most of the chapter describes the effect of CUNY 
Start by presenting student progress using three primary academic outcomes: enrollment, pro- 
gress through developmental education, and college-level credit accumulation. The chapter also 
contains an exploratory discussion of CUNY Start’s effects on student learning, as approximated 
by scores on standardized exit tests in math, reading, and writing. Next, the chapter describes 
whether the effects on the three primary academic outcomes are different among different types 
of students (for example, those referred to developmental education courses in all three subject 
areas compared with those with fewer developmental education requirements). The estimation 
methods for the analyses are described in Appendix E. 


At the time this report was written, data for the full sample of students were available for 
the program semester and the start of the following semester. For most students randomly as- 
signed to the program group, the program semester was the first and only semester they partici- 
pated in CUNY Start (as nonmatriculated students). Most students randomly assigned to the con- 
trol group enrolled in non-CUNY Start courses at CUNY colleges (as matriculated students). 
Because only a limited amount of follow-up data are currently available, these analyses only offer 
a preview of CUNY Start’s effect on student academic outcomes. 


CUNY Start’s Effects on Student Academic Outcomes 


The main analyses in this section compare the academic outcomes of all students in the program 
group with the academic outcomes of all students in the control group. Box 3.2 provides details 
on how the academic outcomes are defined. 


‘Before any analyses were conducted, an analysis plan was written to prespecify which analyses would be 
conducted and which of these would be considered the main indicators of CUNY Start’s success — these are 
called “confirmatory” analyses. The analysis plan specifies that all confirmatory analyses are based on students’ 
academic outcomes at the end of the postprogram semester and beyond. The analyses presented here are con- 
sidered exploratory; they provide information about the program’s effects but will not be the main ones used to 
assess CUNY Start’s success in this evaluation. 
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Box 3.2 
Definitions of Outcome Measures 


This chapter examines CUNY Start’s effects on three types of outcomes: enrollment, college 
readiness, and credits attempted and earned. 


Enrollment: Students have a variety of course offerings and programs available to them at 
CUNY, including CUNY Start, developmental education courses, and college-level courses. 
Students can also take immersion programs and workshops (most of which are intended for 
matriculated students, though not all). Students in CUNY Start who maintained attendance 
through the enrollment cut-off date (typically two to three weeks after the start of the program) 
are referred to as “participating in CUNY Start (as nonmatriculated students).” Students enrolled 
in developmental education courses or college-level courses who remained registered in at least 
one course through the census date (the twenty-first day of classes) are referred to as “enrolling 
in any non-CUNY Start courses (as matriculated students).” Because relatively few students 
enrolled in immersion programs and workshops and data on their participation is incomplete, 
they are not reported separately and are also considered as “enrolling in any non-CUNY Start 
courses (as matriculated students).” The outcome “enrolled at a CUNY college” is a catchall that 
includes students who participated in CUNY Start (as nonmatriculated students) or enrolled in 
any non-CUNY Start courses (as matriculated students). 


College readiness: A student who is eligible to take at least one college-level course in math, 
reading, or writing is considered to be “college-ready” in that subject area. At the time of the 
study, students were generally required to demonstrate college readiness by having high enough 
scores on the SATs, on the New York State Regents exams, on the assessments they took before 
enrolling (“placement tests”), or on the final exams they took at the end of a developmental 
course sequence or CUNY Start (“exit tests”). In developmental courses, students had to pass 
the course in addition to the final test. More details on these tests are available in Appendix B. 
Some colleges also offered “corequisite” remediation courses, which combined developmental 
education and college-level material; students who passed any college-level courses (including 
corequisite courses) are considered college-ready. Additionally, some colleges offered alterna- 
tive pathways (for example statistics instead of algebra in math); students with passing scores at 
the end of any developmental course sequences (including alternative pathways) are also con- 
sidered college-ready. 


Credits attempted and earned: This report focuses specifically on college-level credits at- 
tempted and earned and does not report on any credits associated with developmental education 
courses. Credits attempted are measured based on the courses in which students were registered 
as of the census date. 


Enrollment in the Program Semester 


Enrollment at a CUNY college is the first measure of academic progress. Table 3.1 shows 
educational enrollment among program and control group students (refer to Box 3.3 for an expla- 
nation of how to read this table). 
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Table 3.1 
Enrollment in the Program Semester 


Program Control 


Outcome (%) Group Group Difference 
Enrolled at a CUNY college 89.0 78.7 10.3 
Participated in CUNY Start (as a 
nonmatriculated student) 80.6 0.6 80.0 
Enrolled in any non-CUNY Start 
courses (as a matriculated student) 17.9 78.2 -60.3 
Enrolled in CUNY Start reading/writing or 
developmental English at a CUNY college 68.8 63.9 48 


Enrolled in CUNY Start math or developmental 
math at a CUNY college 75.1 57.5 


Sample size (total = 3,835) 2,997 838 


17.6 


KKK 


KKK 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random 


assignment data, data from CUNY's Institutional Research Database, and CUNY Start 


program participation data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random 


assignment blocks. 


Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 


percent. See Appendix E for details on the impact-estimation model. 


In the program semester, 89 percent of program group students and 79 percent of control 
group students enrolled at a CUNY college (that is, participated in CUNY Start as nonmatricu- 
lated students or enrolled in any non-CUNY Start courses as matriculated students), a difference 
of 10 percentage points.” In other words, one effect of CUNY Start is that it encouraged more 
students to enroll in education. Students in the study were not mandated to enroll in the program 
to which they were randomly assigned. Students assigned to the program group had the option of 


?As discussed in Chapter 1, CUNY Start is offered through the Continuing Education division of a college; 
students participating in CUNY Start do not take college courses while participating in CUNY Start. They are 
considered enrolled at that CUNY college as nonmatriculated students. While most study students who enrolled 
in CUNY colleges enrolled in one of the four study colleges, a small proportion enrolled at a different CUNY 
college. Enrollment at any CUNY college is reflected here. 
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P-Value 


<0.001 


<0.001 


<0.001 


0.003 


<0.001 


Box 3.3 
How to Read the Effect Tables in This Report 


The tables in this chapter that show program effects use the format excerpted below. The first 
row shows that 56.8 percent of students randomly assigned to the program group and 24.7 per- 
cent of students randomly assigned to the control group were college-ready in math at the end 
of the program semester. The difference of 32.1 percentage points is the estimated effect of the 
option to participate in CUNY Start. (See Box 2.1 in Chapter 2 for a more detailed explanation 
of estimated effects.) That is, the offer of CUNY Start caused an estimated 32.1 percentage 
points more program group students to become college-ready in math than would have been the 
case had they not been offered CUNY Start. Moreover, this estimate is statistically significant 
at the 1 percent level (the p-value), indicated by three asterisks. That is, if CUNY Start had no 
true effect, there is less than a 1 percent chance that the estimated effect would be 32.1 percent- 
age points or larger. Two asterisks indicate a p-value of 5 percent, and one asterisk indicates a 
p-value of 10 percent. 


Academic Outcomes for Full Sample 


Program Control 
Outcome (%) Group _ Group __Difference P-value 


College-ready in math at the end of the 


program semester 56.8 24.7 32:1, 3 <0.001 
Sample size (total = 3,835 2,997 838 


participating in CUNY Start, but were not required to do so. Most program group students (81 
percent) did participate in CUNY Start. Eighteen percent of the program group enrolled in non- 
CUNY Start courses during the program semester. (A portion of this 18 percent overlaps with the 
81 percent who participated in CUNY Start.)° Similarly, control group students were not required 
to enroll in non-CUNY Start courses, but they were generally restricted from participating in 
CUNY Start for three semesters. Seventy-eight percent of control group students enrolled in non- 
CUNY Start courses. Fewer than 1 percent of control group students participated in CUNY Start 
during the program semester; that is, a small number of control group students were inadvertently 
allowed to participate in CUNY Start.4 


3Because of the way CUNY Start was aligned with the academic terms at the colleges (see Box 3.1 for 
details), a portion of this 18 percent matriculated after successfully completing CUNY Start. 

4These students are included in the analyses and still considered control group members. This kind of “cross- 
over” is common in randomized trials. In this case, the rate is so low that it does not compromise the findings in 
any substantial way. 
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More program group students than control group students took developmental education 
courses in the subject areas where they required remediation. Students could have worked on 
completing their developmental education requirements by participating in CUNY Start or taking 
a developmental education course or program (such as an immersion program or a developmental 
education workshop course) at a CUNY college. Sixty-nine percent of program group students 
participated in some type of course or intervention to meet a developmental education require- 
ment in reading or writing, compared with 64 percent of the control group.° The difference in 
math is even more striking: 75 percent of the program group participated in math remediation 
compared with 58 percent of the control group. 


Students in the program group may have enrolled in college (and more specifically a 
developmental education course or program) at a higher rate than control group students for sev- 
eral reasons. First, as discussed in Chapter 1, after they went through random assignment students 
assigned to the program group were given a checklist of the steps they need to complete to sign 
up for CUNY Start. Control group students, on the other hand, often had to go to another location 
at the college to meet with advisers and get information on how to register for classes. Second, it 
is possible that CUNY Start may have been more appealing to students because it offered them 
the possibility of making substantial progress through developmental education at a low up-front 
cost and without dipping into their financial aid.° This appeal could serve to increase college 
enrollment. 


CUNY Start’s effects on college readiness and college-level credit accumulation pre- 
sented in this chapter compare these outcomes for the full sample of students, not just those who 
participated in CUNY Start (as nonmatriculated students) or enrolled in non-CUNY Start courses 
at CUNY colleges (as matriculated students). Consequently, these analyses assess the effects of 
students being given the option of CUNY Start. The evaluation focuses on effects defined in this 
way for three reasons. First, the approach incorporates the meaningful effect on enrollment — 
which the option of CUNY Start causes — into its effects on future outcomes. Second, analyzing 
academic outcomes for the full sample of students provides information on the progress a group 
of students can make when given the option of participating in CUNY Start. This information can 


°As shown in Table 2.5, on the student survey more control group students reported taking English classes 
than program group students, a finding that contradicts this finding derived from transcript data. There are two 
possible reasons for the discrepancy. First and most important, students were asked on the survey whether they 
enrolled in any English class, including college-level English, whereas the transcript measure shown in Table 
3.1 excludes students who enrolled in college-level English courses. Second, on the survey students may have 
reported enrollment in developmental education English programs that do not appear in the transcript data. 

®It is also true that the CUNY Start program generally requires more in-class time than a typical control 
group student’s course load at a CUNY college during a single semester, which could lead one to expect fewer 
program group members to participate. However, all of the students in the study — program group students and 
control group students — were considering participating in CUNY Start when they were randomly assigned, so 
they may have already determined how they could fit CUNY Start in with their other commitments. 
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be valuable because CUNY Start is offered (not mandated) as one of many developmental edu- 
cation options available to students at CUNY. Showing outcomes for the full program group, 
then, provides a meaningful assessment of how a group of students may perform when CUNY 
Start is one of the options available to them. Finally, comparing the academic outcomes of the 
full program group with those of the full control group ensures a comparison of two groups that 
were not systematically different at the start of the study. For ease of exposition, throughout the 
remainder of this chapter the results will be discussed as the effects of CUNY Start though they 
are technically the effects of being given the option of CUNY Start. 


The positive 10 percentage point impact on enrollment in the program semester has im- 
plications for the interpretation of the estimated effects on subsequent academic outcomes, such 
as completion of developmental education and credit accumulation. Effects on these outcomes 
may be due, at least in part, to the increased enrollment of program group students rather than to 
the specific structural, curricular, and pedagogical reforms that are the core of CUNY Start.” The 
effects on these later outcomes are important regardless of the specific mechanism that causes 
them. However, the effects on these outcomes should be interpreted differently if they are pri- 
marily a result of inducing more students to enroll rather than primarily a result of the core com- 
ponents of CUNY Start. This issue is discussed in detail in supplemental analyses in Appendix 
G.* These supplemental analyses strongly suggest that the observed effects on completion of de- 
velopmental education and college credit accumulation are not simply a result of the enrollment 
effect. 


Academic Progress by the End of the Program Semester and Beyond 


To assess whether CUNY Start meets the short-term goal of helping students make sub- 
stantial progress through their developmental education requirements before they matriculate, the 
following academic outcomes were measured at the end of the program semester: the number of 
subject areas (math, reading, and writing) where students were “college-ready,” whether students 
were college-ready in each subject area separately, and college-level credits attempted and 
earned. Table 3.2 shows program and control group outcomes, along with the estimated effect of 
CUNY Start on these outcomes at the end of the program semester and the beginning of the first 


"It may also be the case that the enhanced instruction and services of CUNY Start is what led more program 
group students to remain engaged with CUNY Start through the enrollment cut-off date, and that the lack of 
these enhancements is what caused some control group students to withdraw from all of their courses before the 
census date. 

8 Appendix G provides estimates of the effect of participating in CUNY Start compared with what otherwise 
would have happened (a “treatment-on-the-treated” analysis) and also of the effect of participating in CUNY 
Start compared with enrolling in any non-CUNY Start courses. 
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postprogram semester.’ For reference, the first panel of the table shows students’ college readi- 
ness overall and in each subject area before they were randomly assigned (and before they expe- 
rienced the program). '° The outcomes by the end of the program semester are shown in the second 
panel. Enrollment measures during the first postprogram semester are presented in the third panel. 


By the end of the program semester, a higher percentage of students in the program group 
were college-ready in more subject areas than in the control group. Before they were randomly 
assigned, just over half of the students in both groups were not college-ready in any subject area 
(also known as being “triple remedial”). By the end of the program semester, 16 percent of pro- 
gram group students and 24 percent of control group students were triple remedial, an estimated 
effect of -8 percentage points. Similarly, a smaller percentage of program group students than 
control group students were college-ready in only one or two subject areas by the end of the 
program semester. Consequently, a large proportion of program group students (38 percent) suc- 
cessfully completed all developmental education requirements and became fully college-ready. 
Some control group students also made progress through their developmental education require- 
ments, with 13 percent becoming fully college-ready. CUNY Start caused an estimated 25 per- 
centage points more students to complete all their developmental education requirements within 
one semester. 


Another way to look at this result is to consider the total number of subject areas where 
students achieved college readiness. Before random assignment, students in both the program and 
control groups were college-ready in an average of 0.65 out of 3 subject areas. By the end of the 
program semester, program group students were college-ready in an average of 1.87 subject areas 
and control group students were college-ready in 1.38 subject areas. On average, then, program 
group students became college-ready in 1.23 additional subject areas, whereas control group stu- 
dents became college-ready in only 0.73 additional subject areas. CUNY Start caused program 
group students to become college-ready in an estimated 0.49 more subject areas than they would 
have otherwise (1.87 minus 1.38). 


Within math, reading, and writing, more program group students than control group stu- 
dents became college-ready by the end of the program semester. CUNY Start had the greatest 
effect in math, with 57 percent of the program group college-ready in math by the end of the 
program semester, compared with 25 percent of the control group. Given that both groups started 


°A table that meets What Works Clearinghouse guidelines for this analysis and subsequent analyses is in 
Appendix F. 

10Table 1.2 shows student developmental education subject areas required before random assignment. In 
Table 3.2, this same measure is presented as college readiness for consistency with the outcome measures (which 
are also shown in terms of college readiness). Though they are described differently, the two tables show the 
same thing. In Table 1.2 about half of the study sample is presented as requiring developmental education in 
three subject areas before they were randomly assigned; similarly, the first panel of Table 3.2 shows that about 
half of program and control group students were college-ready in zero subject areas before random assignment. 
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with very low college-readiness rates in math (between 5 percent and 6 percent), this gain is par- 
ticularly substantial. Program group students also made more progress in reading and writing than 
control group students. At the start of the study, about 36 percent of students in both the program 
and control groups were college-ready in reading and about 23 percent of students of students in 
both groups were college-ready in writing. Substantial percentages of control group students 
made progress in both areas during the program semester, with 62 percent college-ready in read- 
ing and 52 percent college-ready in writing by the end of the program semester. But even greater 
percentages of program group students made progress in those areas during the same period, as 8 
percentage points more program group students became college-ready in reading and 9 percent- 
age points more became college-ready in writing. 


While program group students, on average, were making substantial progress through 
developmental education, control group students were also making academic progress by at- 
tempting and earning college-level credits. As discussed earlier, most control group students en- 
rolled innon-CUNY Start courses as matriculated students and could take a mix of developmental 
education and college-level courses. During the program semester, control group students there- 
fore had the possibility of earning more college-level credits than program group students. The 
second panel of Table 3.2 shows college level credits attempted and earned by both groups. As 
expected, control group students attempted nearly 3 more college-level credits, on average, than 
did program group students. Much of this lead in credits attempted is maintained in credits earned. 
Control group students earned about 2.4 credits, on average, by the end of the program semester. 
A small proportion of program group students also earned college-level credits. These students 
may have opted to enroll in non-CUNY Start courses instead of participating in CUNY Start, or 
may have enrolled in non-CUNY Start courses as matriculated students in intersessions, after 
completing CUNY Start. They bring the program group average to 0.6 college-level credits. The 
estimated difference in college-level credit accumulation caused by CUNY Start is -1.9 credits. 
To put this finding in context, consider that most associate’s degree programs require 60 college- 
level credits. To earn an associate’s degree within three years (six semesters), therefore, students 
must earn at least 10 college-level credits per semester; the estimated effect of CUNY Start rep- 
resents just under 20 percent of 10 college-level credits. 


Longer-term data are needed to gauge the overall success of CUNY Start and learn 
whether this short-term trade-off of making progress through developmental education in place 
of earning college credits benefits students over time. Currently, only educational participation 
measures are available for the first postprogram semester; these measures are presented in the last 
panel of Table 3.2. In that semester, 5 percentage points more students in the program group than 
in the control group enrolled at CUNY colleges. Most students in the program group enrolled in 
non-CUNY Start courses (as matriculated students), with fewer than 5 percent of program group 
members participating in CUNY Start in the first postprogram semester. This finding is somewhat 
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Table 3.2 


Educational Achievement 


Outcome 


Before random assignment 
Number of college-ready subject areas (%) 


Zero 
One 
Two 


Three (fully college-ready) 
Number of college-ready subject areas (out of 3) 


College-ready in the subject area (%) 
Math 
Reading 
Writing 


End of the program semester 
Number of college-ready subject areas (%) 


Zero 
One 
Two 


Three (fully college-ready) 
Number of college-ready subject areas (out of 3) 


College-ready in the subject area (%) 
Math 
Reading 
Writing 


College-level credits attempted 


College-level credits earned 


60 


Program 


Group 


51.2 
33.4 
15.2 

0.2 


0.64 


5.4 
36.6 
22.3 


16.1 
18.3 
27.7 
37.9 


1.87 


56.8 
69.7 
61.0 


0.8 


0.6 


Control 


Group 


51.0 
33.2 
15.4 

0.4 


0.65 


5.8 
35.7 
23.6 


24.2 
26.7 
36.1 
13.0 


1.38 


24.7 
61.6 
51.6 


3.6 


2.4 


Difference 


0.2 
0.2 
-0.2 
-0.2 


-0.01 


-0.3 
0.9 
-1.3 


-8.1 
-8.4 
-8.4 
24.9 


0.49 


32.1 
8.0 
9.4 


-2.7 


-1.9 


KKK 


KKK 


KKK 


KKK 


P-Value 


0.919 
0.923 
0.894 
0.415 


0.790 


0.704 
0.642 
0.428 


<0.001 
<0.001 
<0.001 
<0.001 


<0.001 


<0.001 
<0.001 
<0.001 


<0.001 


<0.001 


(continued) 


Table 3.2 (continued) 


Program Control 


Outcome Group Group Difference P-Value 


Postprog ram semester 


Enrolled at a CUNY college (%) 69.4 64.3 5.1 *** 0.006 
Participated in CUNY Start (as a 
nonmatriculated student) 3.9 0.5 3.4 *** <0.001 


Enrolled in any non-CUNY Start 
courses (as a matriculated student) 65.9 63.9 2.0 0.288 


Enrolled in CUNY Start reading/writing or 
developmental English at a CUNY college (%) 22.3 30.3 -8.0 *** <0.001 


Enrolled in CUNY Start math or developmental 


math at a CUNY college (%) 19.8 31.9 -12.2 ** <0.001 
College-level credits attempted 7.0 5.2 18 ** <0.001 
Sample size (total = 3,835) 2,997 838 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY's Institutional Research Database, CUNY Start program participation data, and test 
data from CUNY's Administrative Data Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random assignment 
blocks. 


Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. See 
Appendix E for details on the impact-estimation model. 


surprising given that at the start of the study, 30 percent of students required developmental edu- 
cation in all three subject areas but intended to participate in CUNY Start’s part-time program, 
which would allow them to become college-ready in, at most, two subject areas. Such students 
are eligible for a second semester of CUNY Start, but most did not participate in the program 
during the postprogram semester. Similar percentages of program and control group students en- 
rolled at CUNY colleges in non-CUNY Start courses (as matriculated students) in the first post- 
program semester. 


In that first postprogram semester, more control group students than program group stu- 
dents enrolled in developmental education courses. Twenty-two percent of the program group 
enrolled in CUNY Start reading/writing or developmental English, compared with 30 percent of 
the control group. Similarly, 20 percent of program group students enrolled in CUNY Start math 
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or developmental math, compared with 32 percent of control group students. Both estimated ef- 
fects are statistically significant and suggest that a portion of the control group may begin to catch 
up to the program group in college readiness over time. 


Program group students attempted 7.0 college-level credits in the first postprogram se- 
mester, on average, compared with 5.2 college-level credits attempted by control group students. 
The difference of 1.8 college-level credits is statistically significant and substantively meaningful. 
However, at this point, it appears likely that program group students will need more than one 
semester to fully catch up with control group students in college-level credits, given the control 
group’s lead of nearly 2 college-level credits. 


This expectation — that program group students will need several semesters to catch up 
to the control group in college-level credits — is consistent with existing research and some pre- 
liminary analyses. For program group students to catch up in a single semester, CUNY Start 
would have to achieve greater effects on college-level credit accumulation than have been de- 
tected even in studies of highly effective programs. For example, even CUNY’s Accelerated 
Study in Associate’s Programs (ASAP), which nearly doubled graduation rates within three years, 
only boosted college-level credit accumulation by 1.9 credits or fewer for any given semester. !! 
In fact, as mentioned in Chapter 1, CUNY expects students who participate in CUNY Start to 
catch up in college-level credits in their fourth semester. Analyses using the first two cohorts of 
the research sample in this evaluation found that, by the end of the first postprogram semester, 
program group students had begun to close the gap in college-level credits earned but had not yet 
caught up with control group students. It will be important to continue to track the study sample 
for at least four semesters to more fully evaluate CUNY Start’s effectiveness. 


CUNY Start’s Effects on Learning 


It is also important to determine whether assignment to CUNY Start causes students to gain 
greater skills in math, reading, or writing. This question is explored here, including a discussion 
of why answering this question requires strong assumptions, and thus the evidence here is con- 
sidered suggestive, rather than conclusive, unlike the results already presented. 


Before random assignment, most students in the study took standardized placement tests 
in math, reading, and writing, administered by CUNY and called CUNY Assessment Tests 


"Scrivener et al. (2015). 
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(CAT).!* Placement tests were administered for the purposes of placing students into the appro- 
priate math, reading, and writing courses (that is, developmental education or college-level 
courses). At the time of the study, students referred to developmental education were required to 
pass a common exit test in each subject area in order to be deemed college-ready.'* These exit 
tests are a measure of student skills (though an imperfect one), and many students took them 
toward the end of the program semester, well after random assignment, giving them enough time 
to acquire new skills in CUNY Start or other developmental education courses. For the most part, 
only a student who enrolled in and completed a developmental education course or program (such 
as CUNY Start) can take an exit test. 


A strong approach to estimating CUNY Start’s effect on learning would be to compare 
the average exit test scores of all program group members with the average exit test scores of all 
control group members. However, not everyone took the exit tests. In fact, CUNY Start caused 
many more program group students to take the exit tests (see Table 3.3). For example, the pro- 
gram group was 37 percentage points more likely to take the math exit test than the control group. 
This difference occurred for several reasons. First, CUNY Start caused more students to enroll in 
college and to attempt math classes. Second, many control group students were enrolled in mul- 
ticourse developmental math sequences lasting multiple semesters, in which students would only 
take the exit test at the end of the sequence. Finally, some control students were in special reme- 
dial programs that have been exempted from giving the exit test. 


Random assignment does not ensure that subsets of students (for example, test takers) 
identified based on decisions or processes that occurred after random assignment (for example, 
taking an exit test) were highly similar in the program group and the control group at the outset 
of the study. Consequently, differences in the average exit test scores of program and control 
group exit test takers cannot confidently be interpreted as being caused by CUNY Start. Differ- 
ences may also be a result of the types of students CUNY Start caused to take the exit tests. 


In fact, in all three subject areas, the placement test scores of students who ultimately 
took exit tests were lower, on average, in the program group than in the control group. That is, 
program group exit test takers began the study less academically prepared than control group exit 
test takers. Stated differently, by inducing more students to attempt the exit tests, CUNY Start 
caused some initially less-well-prepared students to take the exit test. This fact should be kept in 


12Some students were exempt for various reasons. At the time of the study, the math and reading placement 
tests were COMPASS tests and the writing test was CUNY’s own test. The terms placement test and exit test are 
those of the authors and may not reflect CUNY’s terminology. 

‘3m mathematics, students took the CUNY Elementary Algebra Final Test (CEAFE), a common depart- 
mental exit test. In reading, students took a COMPASS test, and in writing they took a CUNY test. In mathemat- 
ics, students were required to achieve a minimum course average in addition to a minimum test score. In this 
report, however, students are considered college-ready in math if they passed the math test. 
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Table 3.3 


Gains in Skills/Knowledge by the End of the Program Semester 


Program Group 


Sample 


Outcome Size 


CUNY Elementary Algebra Final Exam (CEAFE) 


Took the test (%) 2,997 
Test score (out of 100) — unadjusted 1,832 
Test score (out of 100) 1,832 


CUNY Assessment Test (CAT) in reading 


Took the test (%) 2,997 
Test score (out of 100) — unadjusted 1,275 
Test score (out of 100) 1,275 


CUNY Assessment Test (CAT) in writing 


Took the test (%) 2,997 
Test score (out of 100) — unadjusted 1,595 
Test score (out of 100) 1,595 
Sample size (total = 3,835) 2,997 


Mean 


61.2 
72.6 
73.0 


42.7 
73.2 
73.3 


53.1 
55.8 
55.9 


Control Group 


Sample 
Size 


838 
198 
198 


838 
254 
254 


838 
299 
299 


838 


Mean 


23.8 
66.0 
62.4 


30.0 
71.1 
70.2 


36.5 
55.2 
54.8 


Difference 


37.4 
6.6 
10.6 


12.8 
2.1 
3.1 


16.6 
0.6 
4:4 


KKK 


aKkk 


ak 


KKK 


ak 


ak 


KKK 


kk 


P-Value 


<0.001 
<0.001 
<0.001 


<0.001 
0.005 
<0.001 


<0.001 
0.183 
0.013 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, and test data 


from CUNY's Administrative Data Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random assignment blocks. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. See Appendix E for 


details on the impact-estimation model. 


Final grades are among those who attempted the test in the program semester. 


Italic type indicates nonexperimental data. 


mind when interpreting the average exit test scores of these two groups of students — it appears 


to favor the control group test takers. 


Table 3.3 summarizes the exit test findings — the first row under each exit test panel 
presents the percentage of all students who took the exit test during the program semester. The 


‘4while program group exit test takers began the study less prepared than control group exit test takers, it is 
unknown how the two groups compare with respect to unobserved characteristics such as tenacity, motivation, 
and study habits. It is primarily for this reason that the evidence of CUNY Start’s effect on skills and knowledge 


is considered suggestive rather than conclusive. 
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second row under each exit test presents the average scores among exit test takers, without taking 
into account any pre-random assignment differences between the two groups. The third row under 
each exit test presents the average scores among exit test takers, after statistically controlling for 
pre-random assignment differences in several observable student characteristics, including place- 
ment test scores. 


In all three subject areas, substantially higher percentages of program group members 
than control group members took the test. Despite starting further behind, on average, CUNY 
Start test takers scored higher on the exit test than control group test takers in all three subject 
areas. This finding is notable, and the difference is especially large in math." 


After adjusting for pre-random assignment placement test scores (and several other stu- 
dent characteristics), program group members still scored higher on exit tests than their control 
group counterparts in math, reading, and writing. In all three cases the adjusted differences are 
larger than the unadjusted differences, reinforcing the point that CUNY Start causes initially less- 
well-prepared students to take the exit test, and that CUNY Start is improving learning as meas- 
ured by the exit tests. 


These findings suggest that CUNY Start, with its intensive focus on math, reading, and 
writing, does enable students to gain greater skills in these subject areas — a conclusion that 
aligns with CUNY Start’s theory of change. Again, however, as described above, control group 
students took and earned more college-level credits, often in other subject areas. The control 
group may therefore have gained skills and knowledge in these other areas that cannot be captured 
by this evaluation. 


CUNY Start’s Effects on Academic Progress Among Subgroups 
of Students 


The findings for the full sample presented in the first section of this chapter demonstrate how 
CUNY Start affects students’ academic progress, on average. However, different types of stu- 
dents may respond differently to CUNY Start or the alternative courses and services offered at 


‘SI effect sizes (not shown in the table), the differences are 0.270, 0.190, and 0.08<, in math, reading, and 
writing, respectively. In K-12 research, there is a wealth of information used to provide context and meaning to 
the magnitude of estimated effects presented in effect size units. In community college research, however, such 
information is virtually nonexistent. One approach Hill, Bloom, Black, and Lipsey (2008) offer to aid in inter- 
preting effect sizes is to compare estimated differences in effect-size units to policy-relevant performance gaps. 
For example, achievement gaps on the placement tests between black and white students in the study sample 
were 0.350, 0.060, and 0.020 in math, reading, and writing, respectively. The differences presented above, in 
other words, are as large as or larger than the achievement gap between black and white students. National 
achievement gaps, CUNY-wide achievement gaps, or both may be more meaningful points of comparison. 
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CUNY. Thus, the effects may vary for different students. Such differences are masked when test- 
ing overall average effects. This section explores the extent to which CUNY Start is more effec- 
tive for particular types, or subgroups, of students. 


Three types of subgroups are of particular interest because there are program-related rea- 
sons to expect CUNY Start’s effects to vary by them. '* The definitions of these subgroups follow; 
the reasons why effects were expected to vary by these subgroups defined in these ways are of- 
fered later. 


Intended CUNY Start program: Before random assignment, students were asked 
whether they intended to participate in the full-time or part-time CUNY Start program if they 
were selected for the program group. This subgroup analysis compares the estimated effects of 
CUNY Start among students who intended to participate in the full-time CUNY Start program 
with the estimated effects among students who intended to participate in the part-time program. ’” 


Number of college-ready subject areas before random assignment: CUNY Start is 
designed to serve students with different combinations of developmental education requirements 
in the three subject areas. This analysis compares CUNY Start’s effects among students who had 
zero college-ready subject areas at the start of the study (who were triple remedial) with its effects 
among those who were college-ready in one or two subject areas. ® 


College: Students were randomly assigned at the college in which they intended to par- 
ticipate in CUNY Start or enroll in non-CUNY Start courses. This subgroup is defined based on 
the college where students were randomly assigned. '® 


‘6These three subgroups were prespecified as confirmatory in the analysis plan mentioned in footnote 1 of 
this chapter that was created before any outcome data were examined. The remaining subgroups are exploratory. 
As noted in footnote 1, however, all outcomes in the current time frame are exploratory. 

7™m the program semester, 76 percent of program group students participated in their “intended” CUNY 
Start program type (representing 95 percent of all program group members who participated in CUNY Start). 
Among students in the control group, 68 percent of students who had intended to participate in the full-time 
CUNY Start program (if assigned to CUNY Start) and 55 percent of students who had intended to participate in 
the part-time program attempted a full-time course load of 12 or more credits. Therefore, this subgroup analysis 
is a weak proxy for comparing (1) the effect of participating in the full-time CUNY Start program rather than 
taking a full-time course load in college with (2) the effect of participating in the part-time CUNY Start program 
rather than taking a part-time course load in college. Instead, this analysis provides useful information about the 
effect of being offered CUNY Start among students interested in the full-time program compared with students 
interested in the part-time program. 

'8 small number of students were identified as having no developmental education requirements at the 
start of the study. These students are excluded from this subgroup analysis. 


‘9Most students either participated in CUNY Start at their college of random assignment or enrolled in non- 
CUNY Start courses at their college of random assignment. 
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In addition to these three subgroups, the variation in CUNY Start’s effects was examined 
by subgroups defined using student demographic characteristics that are known to be associated 
with gaps in educational achievement, including gender, age, race/ethnicity, prior education, and 
term of enrollment (fall or spring). 


This section presents the results of subgroup analyses for the following outcomes: edu- 
cational participation, college readiness, and college-level credit accumulation. 


Variation in Effects on Enrollment 


Table 3.4 shows the subgroup results for enrollment, an outcome for which CUNY Start’s 
effects were not expected to vary. In every single subgroup of students examined, CUNY Start is 
estimated to have increased enrollment, and among half of the subgroups the effects are statisti- 
cally significant. Thus, any differences in enrollment effects by subgroup (for example, triple 
remedial students compared with students with one or two remedial requirements) reflect effects 
on enrollment that are all positive, but that are of different sizes. 


The estimated effects on enrollment are similar regardless of students’ intended CUNY 
Start program (full-time or part-time) or number of college-ready subject areas before random 
assignment. However, the effect varies depending on which college the student intended to attend. 
The effect on enrollment was largest at College D, which also had the lowest control group en- 
rollment. At College C, however, the enrollment effect was small and not statistically significant; 
this college also had the highest control group enrollment. Colleges A and B fall between the 
other two. 


The last five panels of Table 3.4 show variations in the enrollment effect by subgroups 
created using student demographic characteristics. Estimated effects are statistically indistin- 
guishable by gender and by prior education level. The enrollment effects are larger among older 
students, Hispanic students, and students who entered the study in a spring semester. The pattern 
among these subgroup findings is that the effect estimates are largest among the subgroups where 
the control group outcome levels are lowest, suggesting program effects are largest among those 
at the greatest risk. 


The next section discusses variation in the program’s effects on progress through devel- 
opmental education. The variation in effects on enrollment may lead to some of the variation in 
effects on academic progress. 
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Characteristic 


Intended CUNY Start program 


Part-time 
Full-time 


Total sample size 


Table 3.4 


Enrollment at CUNY Colleges, 


by Subgroup 
Enrollment (%) P-Value 
Sample Program Control for 
Size Group Group Difference Difference 
1,480 87.4 75.3 12.1 *** <0.001 
2,355 90.0 80.7 9.3---48* <0.001 


3,835 


Number of college-ready subject areas at random assignment 


Zero 
One or two 


Total sample size 


Intended college 
College A 
College B 
College C 
College D 


Total sample size 


Gender 
Female 
Male 


Total sample size 


Age 
19 or younger 
20 to 23 
24 or older 


Total sample size 


1,935 88.6 76.5 12:1) ¥** <0.001 

1,859 89.6 81.4 8.2 *** <0.001 

3,794 

1,167 88.9 79.8 9.1 ** <0.001 
440 92.2 83.2 9.1 *** 0.008 
632 87.7 85.0 2.7 0.405 

1,596 88.8 74.0 148 ** <0.001 

3,835 

1,840 88.7 79.8 9.0 ** <0.001 

1,434 89.4 77.8 T1:5: <0.001 

3,274 

1,837 92.3 85.1 EQ) ORE <0.001 

1,153 87.6 76.7 110 ** <0.001 
845 84.1 66.0 18.14 *** <0.001 

3,835 
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P-Value for 
Differential 
Estimated 
Effects 


0.316 


0.136 


0.013 tt 


0.362 


0.010 tt 


(continued) 


Table 3.4 (continued) 


P-Value for 
Enrollment (%) P-Value Differential 
Sample Program Control for Estimated 
Characteristic Size Group Group Difference Difference Effects 
Race/ethnicity* 0.003 ttt 
Hispanic 1,391 90.8 74.2 16.6 *** <0.001 
White 224 88.4 82.3 6.1 0.294 
Black 985 87.3 80.6 68 ** 0.010 
Asian or Pacific Islander 330 88.9 86.2 2.8 0.570 
Other? 249 84.9 83.5 1.3 0.815 
Total sample size 3,179 
Prior education 0.523 
High school diploma 2,641 90.2 81.3 9.0 *** <0.001 
High school equivalency 497 83.3 71.4 12:0: *** 0.007 
Total sample size 3,138 
Cohort 0.011 = tt 
Fall 1,556 89.5 83.2 6.3  *** 0.002 
Spring 2,279 88.7 75.7 13.0 ** <0.001 
Total sample size 3,835 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY's Institutional Research Database, CUNY Start program participation data , and test 
data from CUNY's Administrative Data Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random assignment 
blocks. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent, ** = 5 percent, * = 10 percent. See Appendix E for details on the impact- 
estimation model. 

A two-tailed t-test was applied to differences in impacts between or among subgroups. Statistical 
significance levels are indicated as: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

aRespondents who selected Hispanic for their ethnicity and chose another racial category are included 
only in the Hispanic category. Respondents who did not select Hispanic for their ethnicity and chose more 
than one racial category are included in the Other category. 

‘Other includes "multiracial," "Native American/Alaskan Native," and other racial/ethnic categories. 
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Variation in Effects on College Readiness by the End of the Program 
Semester 


CUNY Start’s effect on college readiness is positive for each subgroup of students. This 
section focuses on the extent to which those effects are particularly large by certain subgroups of 
students. 


CUNY Start’s effect on college readiness was expected to be greater than average among 
students who intended to participate in the full-time CUNY Start program. By participating in the 
full-time program, students have the possibility of becoming college-ready in all three subject 
areas, whereas students in the part-time program can become college-ready in only one or two 
subject areas. Additionally, the full-time program requires nearly twice as many class hours per 
week (up to 26.5 hours) as the part-time program (13.5 hours), and that additional class time was 
expected to yield greater progress through developmental education. 


Triple-remedial students can complete developmental education requirements in up to 
three subject areas, compared with just one or two subject areas for students who were already 
college-ready in one or two subject areas before random assignment. Thus, triple-remedial 
students were also expected to experience greater-than-average effects on progress through 
developmental education. Because triple-remedial students are disproportionately represented in 
the full-time CUNY Start program, these two factors (that is, being triple remedial and intending 
to participate in the full-time CUNY Start program) are related. 


Finally, effects on college readiness were expected to vary by college due to differences 
in the types of students attending the colleges (including the amount of remediation they re- 
quired), the program’s implementation at the colleges, and the alternative services available to 
students at the colleges. 


Table 3.5 shows the results of the subgroup analyses with respect to the average number 
of subject areas in which students were college-ready by the end of the program semester. As 
predicted, CUNY Start’s effect on college readiness is larger among students who intended to 
participate in the full-time CUNY Start program than it is among those who intended to partici- 
pate in the part-time program. By the end of the program semester, program group students who 
intended to participate in the full-time CUNY Start program were college-ready in 1.87 subject 
areas out of 3 on average, an estimated 0.64 subject areas more than control group students who 
intended to participate in the full-time program. This effect is more than twice as large as the 
effect among students who intended to participate in the part-time program (0.27 subject areas). 
Similarly, as expected, CUNY Start’s effects on college readiness are larger among triple-reme- 
dial students (0.62 subject areas) than among students who were already college-ready in one or 
two subject areas (0.37 subject areas). 
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Table 3.5 


Number of College-Ready Subject Areas (Out of 3) by the End of the 


Characteristic 


Intended CUNY Start program 


Part-time 
Full-time 


Total sample size 


Number of college-ready subject areas at random assignment 


Zero 
One or two 


Total sample size 


Intended college 
College A 
College B 
College C 
College D 


Total sample size 


Gender 
Female 
Male 


Total sample size 


Age 
19 or younger 
20 to 23 
24 or older 


Total sample size 


Program Semester, by Subgroup 


Size 


1,480 
2,355 


3,835 


1,935 
1,859 


3,794 


1,167 
440 
632 

1,596 


3,835 


1,840 
1,434 


3,274 


1,837 
1,153 
845 


3,835 


Average Number of 


Group 


1.87 
1.87 


1.47 
2.30 


2.01 
1.97 
1.92 
1.74 


1.90 
1.84 


2.04 
1.73 
1.72 


Group Difference 
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Sample Program Control 


1.60 
1.24 


0.85 
1.93 


1.58 
1.09 
1.59 
1.22 


1.48 
1.28 


1.47 
1.27 
1.32 


0.27 
0.64 


0.62 
0.37 


0.43 
0.88 
0.33 
0.52 


0.42 
0.57 


0.58 
0.46 
0.39 


College-Ready Subject Areas 


KKK 


KKK 


for 


Difference 


<0.001 
<0.001 


<0.001 
<0.001 


<0.001 
<0.001 
<0.001 
<0.001 


<0.001 
<0.001 


<0.001 
<0.001 
<0.001 


P-Value for 
P-Value Differential 


Estimated 
Effects 
<0.001 ttt 
<0.001 ttt 
<0.001 ttt 

0.060 Tt 
0.090 Tt 


(continued) 


Table 3.5 (continued) 


Average Number of P-Value for 
College-Ready Subject Areas P-Value Differential 
Sample Program Control for Estimated 
Characteristic Size Group Group Difference Difference Effects 
Race/ethnicity* 0.642 
Hispanic 1,391 1.91 1.36 0.55 *** <0.001 
White 224 2.01 1.69 0.32 ** 0.035 
Black 985 1.78 1.30 0.48 ** <0.001 
Asian or Pacific Islander 330 1.93 1.50 0.43 *** 0.002 
Other? 249 1.53 1.14 039 0.003 
Total sample size 3,179 
Prior education 0.245 
High school diploma 2,641 1.91 1.39 0.51 *** <0.001 
High school equivalency 497 1.75 1.36 0.39 *** <0.001 
Total sample size 3,138 
Cohort 0.881 
Fall 1,556 1.94 1.44 0.49 ** <0.001 
Spring 2,279 1.83 1.33 0.50 *** <0.001 
Total sample size 3,835 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY's Institutional Research Database, and test data from CUNY's Administrative Data 
Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random assignment 
blocks. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent, ** = 5 percent, * = 10 percent. See Appendix E for details on the 
impact-estimation model. 

A two-tailed t-test was applied to differences in impacts between or among subgroups. Statistical 
significance levels are indicated as: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

aRespondents who selected Hispanic for their ethnicity and chose another racial category are 
included only in the Hispanic category. Respondents who did not select Hispanic for their ethnicity and 
chose more than one racial category are included in the Other category. 

bOther includes "multiracial," "Native American/Alaskan Native," and other racial/ethnic categories. 
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The third panel of Table 3.5 shows progress through developmental education by col- 
lege. The effects vary substantially by college, ranging from 0.33 to 0.88 subject areas. 
College B (with the largest effect estimate) is the only college that did not offer the part- 
time CUNY Start program, and compared with the other three colleges, College B 
served a much higher proportion of triple-remedial students (about 80 percent of stu- 
dents). As discussed above, both characteristics were found to be related to larger-than- 
average effects on progress through developmental education. 

The remaining panels of Table 3.5 show the estimated effects on progress through devel- 
opmental education across several student demographic characteristics. CUNY Start’s effects 
were larger among male students than among female students. The effects were also larger among 
younger students than older students, the opposite of the variation of the effect on enrollment 
discussed earlier.”° Notably, the effect on progress through developmental education did not vary 
by race/ethnicity, prior education, or semester of enrollment, characteristics that are associated 
with educational achievement gaps. 


Variation in College-Level Credit Accumulation 


CUNY Start’s effect on initial college-level credit accumulation is negative for each sub- 
group of students and at all institutions examined. This section focuses on the extent to which 
those effects are more or less negative for certain subgroups of students. Because program group 
students were expected to attempt and earn zero college-level credits, the expectations for the 
control group dictated the hypotheses concerning which subgroups would fall farthest behind. 


There was no clear-cut hypothesis about whether intending to participate in the full-time 
program would influence CUNY Start’s effects on college-level credit accumulation.*! The pro- 
gram’s effects were expected to be larger than average among students who were college-ready 
in one or two subject areas before random assignment, because when those students were assigned 
to the control group, they were expected to register for more college-level courses than triple- 
remedial students. As with college readiness, college-level credit accumulation can be expected 
to vary by college due to differences in the student populations and the services the colleges of- 
fered. Results from the subgroup analyses on CUNY Start’s effect on college-level credit accu- 
mulation are shown in Table 3.6. 


This difference may be due in part to the fact that, among program group students, a smaller proportion of 
older students enrolled than younger students. 

210n the one hand, it was hypothesized that students who intended to participate in the full-time CUNY 
Start program and were assigned to the control group would register for more credits than students who intended 
to participate in the part-time program and were assigned to the control group. This expectation would lead one 
to predict that CUNY Start’s effects on college-level credits earned would be larger than average among students 
who intended to participate in the full-time program. On the other hand, students intending to participate in the 
part-time program were also expected to have more college-level classes available to them, because they had 
fewer developmental education requirements at the start of the study than full-time students did. 
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In the absence of CUNY Start, students who intended to participate in the part-time 
CUNY Start program earned more college-level credits than students who intended to participate 
in the full-time program (as shown in the first panel). Control group students who intended to 
participate in the part-time CUNY Start program eared 3.1 college-level credits, on average, 
compared with 0.8 earned by similar program group students, an estimated effect of -2.3 credits. 
Among those intending to participate in the full-time CUNY Start program, control group stu- 
dents earned 2.1 college-level credits on average, compared with 0.4 earned by those in the pro- 
gram group, an estimated effect of -1.6 credits. The result occurred, in part, because a large per- 
centage of control group students who intended to participate in the part-time CUNY Start 
program actually registered for a full-time course load when assigned to the control group. 


Taken together, the results indicate that even though CUNY Start had similar effects on 
enrollment by intended participation intensity (that is, by intent to participate in the full-time or 
part-time CUNY Start program), students who intended to participate in the full-time program 
saw greater gains in college readiness and smaller losses of college-level credit accumulation than 
students who intended to participate in the part-time program. Thus, in the short term, CUNY 
Start appears more promising for students who intended to participate in the full-time CUNY 
Start program. 


CUNY Start also had different effects on college-level credit accumulation among triple- 
remedial students than it did among students with fewer developmental education requirements. 
As expected, students in the program group earned few college-level credits regardless of their 
college readiness before random assignment. However, students in the control group earned more 
college-level credits if they were college-ready in one or two developmental subject areas when 
they were randomly assigned than they did if they were college-ready in none. CUNY Start there- 
fore appears to be quite favorable for triple-remedial students in the short term. The story is more 
complex for students who were college-ready in one or two subject areas: The program’s effects 
on progress through developmental education are positive, but these students missed the oppor- 
tunity of earning a larger-than-average number of college-level credits. 


The estimated effects on college-level credit accumulation also vary substantially by col- 
lege. Most of the variation is caused by differences in credit accumulation among the control 
group students at the colleges, as program group students generally earned very few college-level 
credits. In the most extreme example, at College C control group students earned an average of 
5.8 college-level credits, which resulted in a 4.6 college-level-credit deficit for program group 
students. 


74 


Table 3.6 


College-Level Credits Earned by the End of the Program Semester, 


by Subgroup 


Average College-Level 


Credits Earned 


Sample Program 
Characteristic Size Group 


Intended CUNY Start program 


Part-time 1,480 0.8 
Full-time 2,355 0.4 
Total sample size 3,835 


Number of college-ready subject areas at random assignment 


Zero 1,935 0.2 
One or two 1,859 0.9 
Total sample size 3,794 


Intended college 


College A 1,167 0.7 
College B 440 0.1 
College C 632 1.3 
College D 1,596 0.4 
Total sample size 3,835 
Gender 
Female 1,840 0.6 
Male 1,434 0.5 
Total sample size 3,274 
Age 
19 or younger 1,837 0.7 
20 to 23 1,153 0.5 
24 or older 845 0.4 
Total sample size 3,835 


Control 


Group Difference 
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3.1 
2.1 


1.4 
3.5 


2.5 
2.1 
5.8 
1.2 


2.5 
2.5 


2.8 
2.2 
1.9 


-2.3 
-1.6 


-1.2 
-2.5 


-1.8 
-2.0 
-4.6 
-0.8 


-1.9 
-1.9 


-2.1 
-1.7 
-1.5 


kkk 


kkk 


kK 


kK 


kkk 


kkk 


kkk 


kkk 


kK 


kkk 


kK 


kK 


KkK 


P-Value for 
P-Value Differential 
for Estimated 
Difference Effects 
0.002. TTT 
<0.001 
<0.001 
<0.001 ttt 
<0.001 
<0.001 
<0.001 ttt 
<0.001 
<0.001 
<0.001 
<0.001 
0.727 
<0.001 
<0.001 
0.018 = tT 
<0.001 
<0.001 
<0.001 


(continued) 


Table 3.6 (continued) 


Average College-Level P-Value for 
Credits Earned P-Value Differential 
Sample Program Control for Estimated 
Characteristic Size Group Group Difference Difference Effects 
Race/ethnicity* <0.001 ttt 
Hispanic 1,391 0.6 2.0 -1.4  ** <0.001 
White 224 0.8 4.0 -3.2 0 * <0.001 
Black 985 0.6 2.6 -2.0 ** <0.001 
Asian or Pacific Islander 330 0.6 3.1 -2.6  *** <0.001 
Other” 249 0.4 2.4 -2.0 *** <0.001 
Total sample size 3,179 
Prior education 0.004 ttt 
High school diploma 2,641 0.5 2.6 -2.0 *** <0.001 
High school equivalency 497 0.6 1.8 “152. *** <0.001 
Total sample size 3,138 
Cohort 0.043 tt 
Fall 1,556 0.8 2.9 -2.* <0.001 
Spring 2,279 0.4 2.1 -1.7 = *** <0.001 
Total sample size 3,835 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY's Institutional Research Database, and test data from CUNY's Administrative Data 
Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random assignment 
blocks. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent, ** = 5 percent, * = 10 percent. See Appendix E for details on the 
impact-estimation model. 

A two-tailed t-test was applied to differences in impacts between or among subgroups. Statistical 
significance levels are indicated as: ttt = 1 percent; Tt = 5 percent; t = 10 percent. 

aRespondents who selected Hispanic for their ethnicity and chose another racial category are 
included only in the Hispanic category. Respondents who did not select Hispanic for their ethnicity and 
chose more than one racial category are included in the Other category. 

bOther includes "multiracial," "Native American/Alaskan Native," and other racial/ethnic categories. 
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Several factors could be influencing this variation in college-level credit accumulation 
across the colleges. First, the four colleges serve different types of students who may have re- 
sponded differently to the mix of non-CUNY Start services available at their colleges. Second, 
the colleges offer students different non-CUNY Start options and, for example, have different 
approaches to student advising. Whether their decisions can be attributed to the types of non- 
CUNY Start services available at College C, or to the advising strategies employed there, control 
group students at this college took substantially more total credits (college-level and developmen- 
tal education) than control group students at the other colleges. In fact, control group students at 
College C attempted by far the most college-level credits and the fewest developmental education 
credits, even though study students at College C did not have the fewest developmental education 
requirements compared with study students at the other colleges. This finding suggests that some- 
thing about regular course enrollment patterns at College C may be influencing the high level of 
college-level credit accumulation among the control group at this college. 


The effects on college-level credit accumulation vary substantially across most of the 
student demographic characteristics and, as seen in the other subgroups, this variation is largely 
associated with fluctuations in control group students’ credit accumulation. CUNY Start’s effects 
on college-level credit accumulation are more negative among younger students than older 
students; coupled with the larger effect on college readiness, this finding suggests that the trade- 
off of focusing on college readiness in exchange for fewer college-level credits is more dramatic 
for younger students than older students. The effect on college-level credit accumulation also 
varied significantly depending on students’ races/ethnicities, prior education, and entering 
semester, and this variation matches expectations based on known student risk factors and 
educational achievement gaps. For example, while all program group students earned roughly 
comparable numbers of college-level credits regardless of their races/ethnicities, white and Asian 
or Pacific Islander students in the control group earned more credits (4.0 and 3.1, respectively) 
than Hispanic, black, or other students (who earned between 2.0 and 2.6 credits). Since there was 
no discernable variation in effects on college readiness by race, and since Hispanic and black 
students fell the least behind in college-level credits, CUNY Start may be able to help Hispanic 
and black students the most. The results for prior education and entering semester tell similar 
stories. 


At this point, the subgroup analyses present an early view of the types of students for 
whom or the types of circumstances in which CUNY Start may work better. As with the main 
results, long-term follow-up data are essential to understand how the trade-off of focusing on 
college readiness in place of short-term college-level credit accumulation will play out. 
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Chapter 4 


Conclusion 


CUNY Start is an innovative approach to addressing the challenges faced by community colleges 
seeking to meet the needs of underprepared students. Many developmental education reforms aim 
to improve students’ experiences in the traditional, multicourse developmental sequence most 
colleges offer. CUNY Start makes broader changes to the student experience than these other 
reforms. The program increases the amount of time spent on developing skills during a single 
semester; uses curricula that align with research-based practices in math, reading, and writing 
instruction for adults and older adolescents; employs teaching methods based on active learning; 
and includes many forms of student academic support. Students only pay $75 for the program, 
including the course materials, and do not use their financial aid. Additionally, CUNY Start re- 
cruits instructors with content and pedagogical knowledge and provides them with hands-on 
training during an apprenticeship period, and with professional development thereafter. 


Summary of Findings 


This evaluation finds that CUNY Start was generally implemented as its designers intended. Math 
instructors followed the curriculum and pedagogy most of the time. They typically taught the 
prescribed content at the pace required, used the assignments and assessments provided, and ap- 
plied the prescribed instructional techniques. Reading/writing instructors also followed the spec- 
ified curriculum and pedagogy, but the reading/writing curriculum had some flexibility built into 
it, with instructors free to add readings and assignments. CUNY Start gave students enhanced 
support: a college success seminar taught by an adviser that focused on developing skills to suc- 
ceed in both the academic and nonacademic aspects of college, advisers with reduced caseloads, 
and additional forms of academic support inside and outside of class. Finally, instructor job ap- 
plicants were evaluated on their preparation to teach reading, writing, or math and on their will- 
ingness to implement CUNY Start’s curriculum and student-centered pedagogy. Most instructors 
participated in the apprenticeship period before teaching their own classes, as expected, and re- 
ceived professional development in the form of workshops sponsored by CUNY Central and in 
the form of comments on their instruction following regular classroom observations. 


CUNY Start differs from the standard developmental education that control group 
students received in its structure, curriculum, pedagogy, student support services, and staffing and 
training. CUNY Start classes met more frequently. The math curriculum differed substantially 
from traditional developmental math courses in a few ways: It was standardized across the 
colleges, focused on providing a conceptual understanding of math, and incorporated academic 
skill-building activities. CUNY Start’s math pedagogy employed more active learning than did 
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non-CUNY Start courses, in which non-CUNY Start instructors typically spent a greater 
proportion of class time lecturing. 


The reading/writing curriculum shared some learning objectives with traditional 
developmental reading and writing courses, but the two remediation approaches had important 
differences. Most notably, developmental reading and developmental writing were integrated into 
a single course for CUNY Start, while at three out of the four study colleges developmental 
reading and developmental writing courses are taught separately. CUNY Start systematically 
combines reading and writing, which reinforces the relationship between the two by using tools 
such as metacognitive logs and discussion journals; this approach allows students to move more 
quickly through their developmental education requirements in these subject areas. Both CUNY 
Start reading/writing and traditional developmental education courses in these subject areas 
tended to take a student-centered instructional approach, though there is some indication that non- 
CUNY Start instructors also spent more class time lecturing. 


Student support services for control group students typically were not integrated closely 
into the classroom experience, so students often needed to apply more initiative to take advantage 
of advising and tutoring services. Furthermore, control group students generally were not required 
to attend college success seminars; for those who did, the topics covered differed depending on a 
student’s college and program. CUNY Start instructors came from a variety of professional 
backgrounds and generally had fewer advanced degrees than non-CUNY Start instructors, a 
higher proportion of whom had PhDs. Non-CUNY Start instructors, however, had less access to 
personalized and sustained professional development. 


The short-term results on CUNY Start’s effects show that the program is meeting its goal 
of enabling students to complete or reduce substantially their developmental education require- 
ments within one semester, particularly in math. At this point, data are only available from the 
end of the program semester and the beginning of the first postprogram semester. By design, 
students in CUNY Start are asked to make a trade-off: to make a greater priority of developmental 
education requirements than college-level credit accumulation in the short term, with the assump- 
tion that they will be better prepared for college before they matriculate and thereby better set up 
for success once in college. It was expected, therefore, that their peers in the control group would 
accumulate more college-level credits in the short term. Indeed, they did. During the program 
semester, students in the control group made less progress in developmental education than pro- 
gram group students and instead earned some college-level credits. 


Across all subgroups examined, this same pattern holds: Program group students became 
college-ready in more subject areas during the program semester while earning fewer college- 
level credits than similar control group students. However, the current evidence suggests that the 
one-semester trade-off may favor particular types of students more than others, including those 
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planning to participate in the full-time CUNY Start program and those who were assessed as 
needing remediation in all three subject areas. 


Implications of the Findings 


Community colleges and policymakers have recognized for many years that develop- 
mental education requirements are among students’ greatest barriers to graduation. Within the 
span of one semester, CUNY Start enabled students to make substantial progress through devel- 
opmental education — more progress than has been seen in most other developmental education 
reforms that have been evaluated in randomized controlled trials. ! It is noteworthy that a prema- 
triculation program like CUNY Start can produce such substantial results on progress through 
developmental education in a single semester. The effects in math are especially striking, since 
developmental math requirements, in particular, prevent many students from earning degrees.” 
Furthermore, by enabling students to make substantial progress in or complete their developmen- 
tal education requirements, CUNY Start can help them maintain their financial aid eligibility by 
allowing them to avoid repeating noncredit courses (which can affect their ability to meet Satis- 
factory Academic Progress [SAP] standards) or use up their available semesters of federal Pell 
Grant funds. 


CUNY Start’s short-term success is also remarkable given that the program actively tar- 
gets students with substantial developmental education requirements. In contrast, other successful 
developmental education reforms that are currently gaining traction, such as corequisite remedi- 
ation (which combines developmental education, college-level courses, and academic support 
into a single course), are often unavailable to students with the lowest levels of skill in a subject 
area.? 


While these results are promising, it is important to consider whether or not they apply 
more broadly. The specific findings reported here apply only to the research sample. The study 
included a particular group of students who were interested in the possibility of CUNY Start. 
Students interested in the full-time program indicated that they could dedicate up to 26.5 hours of 
time to classes per week, far more hours than are spent in class by many community college 
students. Even students interested in the part-time program indicated that they could dedicate 13.5 
hours to classes per week. Some students in the control group, on the other hand, who are relying 
on standard developmental education courses, may end up spending more total time on their de- 
velopmental education across multiple semesters if they need to take more than one course in a 


'See, for example, Logue, Watanabe-Rose, and Douglas (2016); Scrivener et al. (2015); and Zachry 
Rutschow and Schneider (2011). 


*Attewell, Lavin, Domina, and Levey (2006). 
3See, for example, Logue, Watanabe-Rose, and Douglas (2016). 
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subject area or have to repeat courses. The concentration of class time may be one of the keys to 
CUNY Start’s sizable effect on progress through developmental education, and yet this same 
aspect may be a barrier to participation for students who are balancing several commitments in 
addition to their education and cannot dedicate the required amount of time in a single semester. 
More research is needed to determine whether the program would be the right fit for the broader 
population of students who require developmental education. 


Furthermore, the study occurred within the particular context of four community colleges 
in the CUNY system. The colleges are located in a large, urban setting and have a racially diverse 
mix of students. Additionally, during the period discussed in this report, CUNY required students 
to pass tests to exit developmental education, while many other community colleges have differ- 
ent policies. It is uncertain whether the barriers to student success and CUNY Start’s approaches 
to addressing these barriers can be applied in other community college contexts. 


Nonetheless, these findings suggest that with the right mix of reforms, students with a 
variety of developmental requirements — including those with substantial requirements in one or 
more subject areas — can make a considerable amount of progress. It is not known whether spe- 
cific components of the CUNY Start model made more of a difference in students’ experience 
and progress than others. One could argue that increased time spent on the developmental disci- 
plines in a compressed time frame may help students focus their learning and gain the subject 
skills more quickly. 


At the same time, the increased intensity of the program cannot be separated from the 
curricular and pedagogical reforms. The sizable effect on the percentage of students who became 
college-ready in math compared with the smaller but still meaningful effect on the percentage of 
students who became college-ready in reading and writing may indicate that certain components 
of the model simply produce more dramatic effects in math. The larger effect on math completion 
may also be because there are greater differences in math curriculum and pedagogy between 
CUNY Start and non-CUNY Start courses, whereas CUNY Start and non-CUNY Start reading 
and writing curriculum and pedagogy shared some techniques and principles. Among members 
of the study research sample, CUNY Start also had more room to produce larger effects in math 
than in reading and writing, as the vast majority of study students (over 90 percent) required at 
least some remediation in math at the start of the study and many may have needed to take more 
than one developmental math class. In contrast, more students were assessed as college-ready in 
reading, writing, or both at the start of the study. 


The student support services are also important to keep in mind, as they affect how stu- 
dents seek out guidance on succeeding in a course or navigating through college. In particular, 
the lower student-to-adviser ratio and integrated student services in CUNY Start allowed staff 
members to pay more attention to individual students and potentially to intervene more quickly 
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when students encountered obstacles. The changes to instructor recruitment and professional de- 
velopment also inevitably inform the experiences students have in the classroom and outside of 
it. 


Though the effects on progress through developmental education are impressive, it re- 
mains an open question how the trade-off of making a greater priority of developmental education 
over college-level credit accumulation in the short term will play out in the long term. The pro- 
gram theorizes that if it can improve students’ college readiness in math, reading, and writing, 
and help them develop as learners before they matriculate, students will be better prepared and 
make greater progress toward their degrees in subsequent terms. Given the limited amount of 
follow-up data currently available in this evaluation, at this point it is only certain that the trade- 
off of developmental education over college-level credit accumulation took place. In subsequent 
semesters, program and control group students may continue to make progress through develop- 
mental education and earn college-level credits. Since CUNY Start compressed the amount of 
time students needed to complete developmental requirements to one semester, it is possible that 
some control group students will begin to catch up to program group students in terms of college 
readiness, while other control group students who put off completing their developmental educa- 
tion requirements may be unable to enroll in additional college-level courses. Follow-up data 
must be collected over a longer period to see whether students in the program group persist in 
college, catch up with the control group in college-level credit accumulation, and, most important, 
surpass them in graduation rates. 


In the larger context of developmental education reform, CUNY Start is one program 
with remarkable short-term results. Understanding whether the trade-offs CUNY Start offers to 
students results in long-term educational gains will be another valuable piece of evidence 
regarding what strategies can help some of the lowest-performing students succeed in their long- 
term academic goals. For example, if CUNY Start’s effects on students’ academic progress are 
sustained through graduation, the program could serve as a useful option for students with 
substantial developmental requirements, particularly triple-remedial students, who currently have 
stubbornly low graduation rates. 


Future Research 


The research collaboration among MDRC, the Community College Research Center, and CUNY 
will continue to share results and guidance for the field from this evaluation. MDRC will follow 
the academic progress of students in this study for at least two years after they enrolled in the 
study to measure their longer-term educational outcomes. The team will also explore CUNY 
Start’s costs and cost-effectiveness, which will be useful in determining the program’s relevance 
outside the CUNY context. A final report presenting longer-term effects and cost-effectiveness is 
scheduled to be published in 2020. In addition, the Community College Research Center will 
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publish two topical papers focusing on CUNY Start’s math curriculum and pedagogy and CUNY 
Start’s staffing and professional development model. Finally, CUNY will develop a toolkit on 
CUNY Start best practices, with a focus on CUNY’s internal use of data for program management 
and evaluation. 
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Appendix A 


Students’ Baseline Characteristics 


Student characteristics at the time of random assignment come primarily from two main data 
sources: test data from the City University of New York’s (CUNY’s) Administrative Data Ware- 
house and student responses on the CUNY Start application. Before random assignment, each 
student was asked to fill out a CUNY Start application. However, several student applications 
were not collected or completed until after random assignment. To avoid bias in the collection of 
application forms after random assignment (for example, collecting more application forms from 
program group members after random assignment than control group members because program 
group members were in CUNY Start and could be tracked), only data from applications collected 
before random assignment were used in any of the statistical analyses presented in this report. As 
a result, data from the CUNY Start application are missing for 14 percent of the study sample. 
Table 1.2, which provides descriptive information on the research sample, shows percentages 
among those with CUNY Start application data, including those sample members who only sub- 
mitted their applications after random assignment, whereas Appendix Table A.1 shows percent- 
ages among the full sample and only includes data from applications collected before random 
assignment. 


Test data from CUNY’s Administrative Data Warehouse are handled similarly, with a 
few differences. Pre-random assignment test data are used to determine students’ developmental 
education requirements at the start of the study. Some students took placement or exit tests in the 
time after they were randomly assigned and before the start of their program semester; again, to 
avoid bias, these test results are considered as follow-up data and not baseline data. Students who 
are missing pre-random assignment test data for a subject area are assumed to require remediation 
in that subject area. 


The goal of random assignment is to have relatively similar students in the study’s pro- 
gram and control groups. A few tests were conducted to confirm whether the program and control 
group students were truly similar. The first of these is shown in Appendix Table A.1: The pro- 
portion of students with particular characteristics is shown for the program and control group, and 
the differences between the groups are tested to see if any are statistically significant. The only 
characteristic with a statistically significant difference (at the 10 percent level) is the number of 
hours worked among those employed (note that the difference between the two groups in em- 
ployment rates was not significant). Given the number of tests that were run, one statistically 
significant finding is not surprising. Second, an omnibus F-test was conducted to see whether 
students’ baseline characteristics were jointly predictive of students’ random assignment status. 
Logistic regression was used for this analysis, where the outcome was students’ random assign- 
ment status and the predictors were the student characteristics. The result from this test was not 
statistically significant, with a p-value of 0.885. Finally, a comparison between the program and 
control groups shows that the magnitude of any of the differences in baseline characteristics is 
small (rarely exceeding 3 percentage points). Taken together, this information demonstrates the 
similarity of the program and control groups at the start of the study. 
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Appendix Table A.1 


Baseline Characteristics of the Evaluation Sample, by Research Group 


Characteristic (%) 


Gender 
Female 
Male 
Missing 


Age 
19 or younger 
20 to 23 
24 or older 


Race/ethnicity* 
Hispanic 
White 
Black 
Asian or Pacific Islander 
Other? 
Missing 


Native language 
English 
Spanish 
Other 
Missing 


Marital and household status 
Married, living with spouse 
Married, apart from spouse 
Unmarried, living with partner 
Unmarried, not living with partner 


Missing 
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Program 


Group 


48.3 
36.7 
15.0 


47.6 
30.4 
22.0 


35.9 
5.9 
25.3 
9.1 
6.4 
17.5 


46.5 
18.6 
19.6 
15.4 


4.9 
1.6 
10.7 
50.0 
32.9 


Control 


Group 


46.2 
40.0 
13.8 


48.6 
28.9 
22.5 


36.9 
5.7 
27.4 
6.8 
7.0 
16.3 


49.0 
17.9 
18.7 
14.4 


4.7 
2.6 
9.0 
50.6 
33.2 


Difference P-Value 


0.216 
2.1 
-3.3 
1.2 


0.691 
-1.0 
1.5 
-0.5 


0.278 
-1.0 
0.2 
-2.1 
2.3 
-0.6 
1.1 


0.620 
-2.6 
0.7 
0.8 
1.0 


0.246 
0.2 
-1.0 
1.7 
-0.6 
-0.3 


(continued) 


Appendix Table A.1 (continued) 


Characteristic (%) 
Household status 

Lives with parents 

Lives away from parents 


Missing 


Expenses 
Parents pay more than half 
Parents do not pay more than half 


Missing 


Children under the age of 18 
Has one or more children under the age of 18 
Does not have any children under the age of 18 


Missing 


Dependents 
Has any children or adults dependent on student 
for financial support 
Does not have any children or adults dependent 
on student for financial support 


Missing 


Financial aid 
Applied for financial aid 
Did not apply for financial aid 


Missing 


Employment 
Is currently employed 
Is not currently employed 


Missing 
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Program 


Group 


65.3 
18.6 
16.1 


32.6 
33.2 
34.2 


9.5 
74.6 
15.8 


12.1 


69.3 
18.5 


65.6 
17.3 
17.1 


41.7 
43.6 
14.7 


Control 


Group 


64.2 
20.4 
15.4 


31.0 
33.2 
35.7 


10.1 
74.4 
15.5 


13.1 


68.7 
18.1 


68.9 
14.9 
16.2 


42.6 
43.6 
13.7 


Difference 


1.1 
-1.8 
0.7 


1.5 
0.0 
-1.5 


-0.6 
0.3 
0.3 


-1.0 


0.6 
0.4 


-3.3 
2.4 
0.9 


-0.9 
0.0 
0.9 


P-Value 
0.480 


0.634 


0.880 


0.740 


0.156 


0.770 
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Appendix Table A.1 (continued) 


Program 
Characteristic (%) Group 
Among those currently employed: 
Number of hours worked per week 
1-10 9.6 
11-20 20.8 
21-30 23.2 
31-40 28.6 
More than 40 4.3 
Missing 13.5 
Family education 
Is the first person in the family to attend college 29.0 
Is not the first person in the family to attend college 53.6 
Missing 17.5 
Highest diploma or degree earned by the parent with 
the most education 
Not a high school graduate 14.2 
High school diploma or equivalent 22.3 
Some college (but not a degree) 9.3 
College degree (AA, BA, MA, PhD) 20.4 
Missing 33.9 
Diplomas and degrees earned® 
High school diploma 68.6 
High school equivalent 13.5 
Occupational or technical certificate 2.6 
Other 3.4 
None of the above 1.4 
Missing 14.7 
Date of high school graduation or equivalent 
During the past year 61.8 
Between one and two years ago 3.7 
Between two and five years ago 3.9 
More than five years ago 5.9 
Missing 24.6 
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Control 


Group 


10.2 
14.4 
27.1 
28.0 

4.1 
16.2 


28.8 
53.8 
17.4 


12.5 
20.7 

9.9 
21.9 
35.1 


69.0 
13.5 
3.4 
4.1 
1.5 
13.7 


62.3 
4.2 
3.6 
5.9 

24.0 


Difference P-Value 


-0.6 
6.4 
-3.9 
0.6 
0.2 
-2.7 


0.2 
-0.3 
0.1 


1.7 
1.6 
-0.6 
-1.5 
-1.2 


-0.4 
0.0 
-0.7 
-0.7 
-0.1 
0.9 


-0.5 
-0.5 
0.3 
0.0 
0.7 


0.101 


0.992 


0.492 


0.771 
0.787 
0.434 
0.533 
0.785 
0.495 


0.937 
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Appendix Table A.1 (continued) 


Program Control 

Characteristic (%) Group Group 
Student risk® 

Traditional student 50.4 52.0 

Nontraditional student 37.9 37.6 

Missing 11.7 10.3 
College attendance 

Has attended college previously 7.2 8.9 

Has not attended college previously 78.1 77.4 

Missing 14.7 13.7 
College enrolled 

College A 30.4 30.4 

College B 11.5 11.5 

College C 16.5 16.5 

College D 41.6 41.6 
Expected enrollment in the coming semester 

Full time 61.4 61.4 

Part time 38.6 38.6 


Number of subject areas requiring developmental education 


Zero (fully college-ready) 0.2 0.4 
One 15.2 15.4 
Two 33.4 33.2 
Three 51.2 51.0 


Highest degree student hopes to achieve 


Some college (but not a degree) 0.4 0.3 
Associate's degree 10.3 10.9 
Bachelor's degree 45.7 45.2 
Postgraduate or professional degree 20.4 21.6 
Missing 23.2 22.1 
Sample size (total = 3,835) 2,997 838 
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Difference P-Value 


-1.6 
0.3 
1.3 


-1.7 
0.7 
1.0 


0.0 
0.0 
0.0 
0.0 


0.0 
0.0 


-0.2 
-0.2 
0.2 
0.2 


0.1 
-0.6 
0.5 
-1.2 
1.1 


0.512 


0.231 


1.000 


1.000 


0.777 


0.844 
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Appendix Table A.1 (continued) 


SOURCES: CUNY Start application data, MDRC random assignment data, and test data from CUNY's 
Administrative Data Warehouse. 


NOTES: All values are weighted to account for random assignment ratios that vary across random 
assignment blocks. 

An omnibus F-test was conducted to see whether students’ baseline characteristics were jointly 
predictive of students’ random assignment status. The results were not statistically significant (p = 
0.885). 

aRespondents who selected Hispanic for their ethnicity and chose another race category are 
included only in the Hispanic category. Respondents who did not select Hispanic for their ethnicity and 
chose more than one racial category are included in the Other category. 

>Other includes "multiracial," "Native American/Alaskan Native," and other racial/ethnic categories. 

Distributions do not add to 100 percent because categories are not mutually exclusive. 

dNontraditional students are defined as those who were 24 or older, who worked 35 or more hours 
per week, who had children, or who had not received a high school diploma and who were not enrolled 
in high school at the time of random assignment. Students are listed as nontraditional if they fit any of 
these characteristics. Students are considered to be "missing" in the nontraditional category if they are 
missing data on two or more of these characteristics and have no other nontraditional characteristic. 
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Appendix B 


Data Sources 


The analyses in this report rely on multiple data sources, described below. 


Chapter 1 


CUNY Start application. Just before students were randomly assigned, they 
were asked to complete the CUNY Start application, which includes demo- 
graphic and other background information. These data are used in Chapter 1 
to describe the evaluation sample. The data are also used in Chapter 3 to define 
subgroups of students. 


Chapter 2 


Interviews and focus groups. In the spring 2015, fall 2015, and spring 2016 
semesters, staff members from the Community College Research Center 
(CCRC) conducted interviews with CUNY Start and non-CUNY Start admin- 
istrators, instructors, advisers, tutors, and students, as well as with the CUNY 
Start professional development staff. In all, CCRC conducted 134 interviews. 
The interviews provided detailed information on the operation of CUNY Start 
and information about the service contrast between CUNY Start and standard 
college courses and services. 


Classroom observations. In the spring 2015, fall 2015, and spring 2016 se- 
mesters, CCRC and MDRC staff members conducted structured observations 
of CUNY Start and standard developmental math, reading, and writing 
courses. In total, the research team observed 23 math classes and 23 read- 
ing/writing classes. Two researchers observed each class and used a structured 
protocol to record course format, pedagogical approaches, curricular compo- 
nents, and student engagement at five-minute intervals. Staff members also 
observed 7 CUNY Start college success seminars. 


CUNY Start professional development observations. In the summer 2015 
and fall 2015 semesters CCRC and MDRC staff members conducted 
observations of CUNY Start professional development training sessions and 
workshops. 


CUNY Start curricula and other documents. The City University of New 
York (CUNY) provided CCRC with documents that describe in detail CUNY 
Start’s math and reading/writing curricula, as well as documents describing 
CUNY Start’s core values. The documents provided detailed information 
about the content and approaches of CUNY Start’s courses. 
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Course syllabi. CUNY Start staff members provided CCRC with syllabi from 
CUNY Start math and reading/writing classes. Departments provided CCRC 
with syllabi from 7 standard developmental math courses across three of the 
colleges in the evaluation and 10 standard developmental reading and writing 
courses across all four of the colleges. The syllabi provided information about 
how the courses were structured, what topics were covered, and assignments 
and expectations for students. 


Student survey. A survey was administered to some students in the evaluation 
sample during their first semester in the evaluation — all of the control group 
members and 67 percent of the program group members.! The survey was ad- 
ministered in the spring 2015, fall 2015, and spring 2016 semesters. It was 
initially administered online; students who did not respond were called on the 
phone. In the end, 75 percent of the students in the targeted sample completed 
the survey — 75 percent of the targeted program group members and 75 per- 
cent of the control group members. As is described in Appendix D, program 
group and control group students who completed the survey were similar. The 
survey covered topics such as sample members’ participation in and experi- 
ences with student services; experiences in math, reading, and writing class- 
rooms; and expectations about and engagement in college. Findings from the 
student survey are used in Chapter 2 to help describe CUNY Start and the 
service contrast between CUNY Start and standard college courses and ser- 
vices. 


Instructor survey. During the spring 2015, fall 2015, and spring 2016 semes- 
ters, a survey was administered to all CUNY Start instructors who had com- 
pleted the apprenticeship semester (66 instructors) and a random sample of 
276 standard developmental education instructors at the four colleges in the 
evaluation.” The survey was initially administered online; instructors who did 
not respond were called on the phone. In the end, 63 percent of the instructors 
who were targeted for the survey completed it — 86 percent of the targeted 
CUNY Start instructors and 58 percent of the targeted standard developmental 
education (or “non-CUNY Start”) instructors. As is described in Appendix D, 


‘Survey fielding started within a few weeks of the end of random assignment. A few control group students 
were not given the survey because informed consent forms had not yet been collected from them. Once informed 
consent forms were obtained it was too late to include these students in the survey, but they are included in the 


outcome analyses in Chapter 3. 


On the survey, 4 instructors identified as CUNY Start instructors and 11 instructors identified as standard 
developmental education instructors said they did not teach courses in the semester in which they were adminis- 


tered the survey. They were excluded from all analyses, samples, and response-rate calculations. 
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because of the low response rate, the survey results for the non-CUNY Start 
instructors should be interpreted cautiously; it is quite possible that non- 
CUNY Start survey respondents are not representative of all non-CUNY Start 
instructors. The survey covered topics such as the instructors’ backgrounds, 
the curricula and teaching methods used in their courses, and their experiences 
with professional development. Findings from the instructor survey are used 
in Chapter 2 to help describe CUNY Start and the service contrast between 
CUNY Start and standard college courses and services. 


Chapter 3 


CUNY student transcript data. CUNY provided MDRC with information 
on the courses students took at the four colleges in the study and all the other 
colleges in the CUNY system. Transcript data from CUNY’s Institutional Re- 
search Database are available for all sample members for the semester after 
random assignment (the “program semester’) and through registration for the 
following semester (the “postprogram semester”). The postprogram semester 
registration data are preliminary and subject to change once final data (with 
grades) become available. Transcript data are used in Chapter 3 to provide a 
detailed look at CUNY Start’s effects on students’ academic outcomes. 


CUNY Start program participation data. CUNY provided MDRC with in- 
formation on sample members’ participation in CUNY Start. These data are 
collected and maintained by the CUNY Office of Research, Evaluation, and 
Program Support, which provides data support to CUNY Start and conducts 
ongoing evaluation of the program. 


Test data. CUNY provided data from its Administrative Data Warehouse on 
various tests that sample members took. Students at CUNY’s colleges can 
demonstrate that they are prepared for college-level courses in a number of 
ways. Before they begin classes, students can take tests administered by 
CUNY in math, reading, and writing (known as the CUNY Assessment Tests, 
or CAT, and referred to in this report as “placement tests”). At the start of the 
study, the math and reading placement tests were COMPASS tests. In October 
2016, CUNY began using ACCUPLACER rather than COMPASS. The writ- 
ing placement test, called the CUNY Assessment Test in writing, was created 
by CUNY. Students can also submit New York Regents exam scores and SAT 
scores to provide evidence of college readiness. Regents exams are statewide 
standardized exams in core high school subjects. Students with high enough 
Regents exam scores or SAT scores are placed directly into college-level 


97 


courses.° Finally, the individual colleges can grant exemptions on a case-by- 
case basis to allow students to take college-level courses in a given subject 
area, though such exemptions are very rarely granted. 


During the period covered in this report, students in developmental education courses 
were generally required to pass tests to demonstrate their proficiency (referred to in this report as 
“exit tests”).4 For reading and writing, the same tests were used for placement and exit. For math, 
the exit test is the CUNY Elementary Algebra Final Exam (CEAFE), created by CUNY. 


Test data are used in Chapter 3 to gauge students’ progress through developmental edu- 
cation and to define subgroups of students. 


3Depending on the version of the Regents exam submitted, CUNY may not only need to see that students 
achieved a high enough score on the exam but may also analyze information from students’ applications to de- 
termine whether they passed certain qualifying high school math classes. 

‘There were a few exceptions. For example, students taking particular workshop classes may not have been 
required to take exit tests. 
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Appendix C 


Implementation Research Methods 


To complement the impact analyses, the implementation research aimed to answer the following 
research questions: 


1. 


2. 


3. 


How is CUNY Start implemented? 
To what degree is there fidelity to the CUNY Start model? 


To what degree is there a service contrast between CUNY Start and the colleges’ 
standard courses and services? 


The implementation research was led by the Community College Research Center 
(CCRC) and conducted by experienced CCRC and MDRC researchers at the four City University 
of New York (CUNY) community colleges participating in the evaluation. The implementation 
research was designed to provide insight into CUNY Start’s institutional context, describe fidelity 
to the model and the service contrast, and generate in-depth and detailed accounts of factors 
influencing fidelity and service contrast. In addition to the instructor survey and student survey 
(discussed in Appendix D), the research team conducted the following three qualitative data- 
collection activities. See Appendix Table C.1 for a summary of implementation research data- 
collection activities. 


1. 


Interviews and focus groups. At each of the four evaluation colleges, researchers 
conducted interviews, focus groups, or both with CUNY Start instructors, non- 
CUNY Start developmental instructors, CUNY Start and non-CUNY Start advisers, 
academic support staff members, college administrators, CUNY Central administra- 
tors, and CUNY Start and non-CUNY Start students. Focus group and interview data 
were collected over three semesters using semistructured protocols that explored im- 
plementation topics in depth. Due to the large number of stakeholders on the control 
side and the smaller number of stakeholders involved with CUNY Start, the research 
team aimed to speak to all CUNY Start stakeholders except for students, who were 
randomly selected. Stakeholders across the control condition were identified by re- 
ferrals from other participants and selected based on their roles and willingness to 
participate. 


Classroom observations. Researchers conducted observations of courses and 
CUNY Start professional development sessions during three semesters of the study 
in order to better understand how the CUNY Start and traditional developmental ed- 
ucation curricula and pedagogy were enacted. Classroom observations were con- 
ducted in a sample of CUNY Start math, reading/writing, and student seminar clas- 
ses. Researchers also observed non-CUNY Start developmental math, reading, and 
writing classes at the four evaluation colleges to understand service contrast. Re- 
searchers identified potential classes based on a list of all classes being taught during 
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Appendix Table C.1 
Implementation Research Data-Collection Totals 


CUNY  Non-CUNY 


Research activity Start Start Total 


Observations 


Math 14 9 23 
Reading/writing 12 11 23 
Seminar 6 - 6 


Instructor and department chair interviews 
Math 10 10 20 


Reading/writing 11 15 26 


Student interviews 
Student focus groups 6 5 11 


Administrator and student success staff member interviews 
Senior-level administrators 15 22 37 


Tutors and other support staff members 15 12 27 


CUNY Central interviews 
Senior-level administrators - - 


Professional development providers - - 


SOURCE: Community College Research Center field research data. 


NOTE: CUNY Central interviewees serve students in both research groups. 


that semester; classes were selected based on instructors’ willingness to participate. 
When possible, researchers selected classes to observe in which the instructor also 
agreed to participate in an interview. Doing so allowed researchers to ask follow-up 
questions about what they observed during the class. Individual instructors were ob- 
served once during the study using an observation protocol created and pilot tested 
by the research team. Researchers kept a running record of classroom activities and, 
through a standardized observation protocol, documented the curricular content, ped- 
agogical practices, and student engagement exhibited in a single class period. At five- 
minute intervals, researchers recorded the content, materials, instructional activity, 
pedagogy, and level of student engagement. The instructional-activities and peda- 
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gogy indicators drew from research on classroom practice and the CUNY Start in- 
structional model. The instructional-activities categories included lesson set-up, in- 
troduction of a new topic, practice of topics, review of topics, assessment, and lesson 
wrap-up. The pedagogy categories included: 


a. 


Instructor leads lecture: Instructor is lecturing with little to no in- 
volvement from students. 

Instructor leads interactive discussion: Instructor is lecturing with fre- 
quent involvement from students. 

Instructor facilitates interactive learning: Instructor sets up a discus- 
sion in which students are the primary speakers. Instructor speaks less 
than 50 percent of the time. 

Instructor initiates small-group work: Instructor breaks students into 
groups with a task. Select this line only if instructor is NOT circulating 
and providing support. 

Instructor provides support to small groups: Instructor circulates 
class to check on each small group and assist them. 

Instructor initiates individual work: Instructor assigns a task for stu- 
dents to work on individually. Select this line only if instructor is NOT 
circulating and providing support. 

Instructor provides support to individuals: Instructor circulates class 
to check on each individual and assist him or her. 

Students present prepared work: Students present work that was pre- 
pared in advance. Includes formal presentations (for example, a culmi- 
nating project) and less formal reporting (for example, groups report on 
their learning after a period of small-group work). Does NOT include 
impromptu student performance (for example, a student comes to the 
board to complete a math problem). 

Pedagogy other: Some other pedagogical practice not indicated in other 
categories. 


Typically, two researchers conducted each observation, with one researcher respon- 
sible for the running record and the other responsible for the time-series protocol. 


Researchers also conducted observations of CUNY Start professional development 
activities. For these observations, researchers used a running record. The professional 
development activities included planning sessions as well as workshops for staff 
members. 
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3. Curricular review. CCRC collected and reviewed curricular materials (for 
example, curriculum guides, lesson plans, syllabi, assignments, assessments, 
and student work), training materials, program descriptions, internal research, 
and policy documents from the CUNY Start central office and from all the 
evaluation colleges related to CUNY Start and CUNY’s regular developmen- 
tal education offerings. Researchers’ aim was to understand the content and 
intent of various curricula as well as other factors that may have an impact on 
policy, practice, and academic outcomes. 


Implementation research analysis was conducted by CCRC and organized according to 
the components of the CUNY Start program and its theory of change, with the goal of document- 
ing the implementation of CUNY Start, assessing whether the program was implemented with 
fidelity to its model, and gauging the service contrast. These components include course offerings, 
academic and other support services for students, and recruitment and training processes for in- 
structors. The student and instructor survey data were analyzed along with interview, focus group, 
and observation data, in order to examine variation between the program and control groups. 


For interviews, focus groups, and the running record from the classroom observations, 
CCRC created a systematic process for managing, coding, and analyzing data. The coding pro- 
cess was intended to organize the interview and focus group data as well as the running-record 
section of the classroom observation data. The codes focused on broader topics within the data 
such as staff and instructor recruitment, pedagogy, curriculum, academic advising, and academic 
behaviors. Researchers were trained to apply codes consistently. The second round of coding 
provided a higher level of detail. 


The analysis of data from interviews, focus groups, and classroom observation running 
records began by examining the data attached to each code to answer the research questions. The 
process was organized by the following categories: curriculum, pedagogy, student support, and 
staffing and training. For example, to understand how the math pedagogy was intended to be 
delivered, the researchers reviewed data collected from professional developers that were coded 
as “pedagogy” and “math.” Researchers then read through the results to identify themes and con- 
ducted additional reviews when necessary. These data were then arranged by subtopic and re- 
searchers refined the themes before selecting illuminating quotes to be used when writing the 
report. 


Survey responses were analyzed according to the construct in question. For example, 
survey items designed to address instructors’ use of questioning were examined among program 
respondents for fidelity and compared between the program and control conditions to illuminate 
any service contrast. In most cases these data affirmed what was found in the qualitative data; 
however, discrepancies existed between instructor and student survey reports. 


104 


Finally, time-series data were coded by activity and length of time spent on the activity. 
These data were aggregated for math and reading/writing and for the program and control groups. 
They enabled the researchers to see whether they had observed the practices and concepts that 
instructors and students noted in their qualitative interviews and focus groups and on the survey. 
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Appendix D 


Student and Instructor Survey Documentation 
and Analyses 


This appendix discusses the student and instructor surveys. For both surveys it provides infor- 
mation on survey fielding and response rates. For the student survey only, the appendix also dis- 
cusses response bias analyses. 


Student Survey 


Survey Fielding and Respondent Sample 


The student survey asked study participants (in the program and control groups) about a 
variety of topics including their experience with math and reading/writing courses and student 
services, as well as their engagement with school. Initially, students were sent multiple email 
messages encouraging them to complete the survey online. Later, students who did not complete 
the survey online were called on the phone to complete it in a computer-assisted telephone inter- 
view. Abt Associates led the fielding of the survey. 


The survey was fielded to a stratified random sample of approximately two-thirds of all 
program group members (Np-surveyea = 1,985) and to all control group members 
(Nc-surveyea = 824).' Fielding occurred during the first semester after a student was randomly 
assigned, starting around midway through the semester and lasting approximately 9 to 11 weeks. 
A total of 2,098 responses were collected (1,480 in the program group and 618 in the control 
group), an overall survey response rate of 74.7 percent; 74.6 percent of program group students 
responded and 75.0 percent of control group students responded. 


Characteristics of Survey Respondents 


Two analyses were conducted to understand how nonresponse might influence the inter- 
pretation of the findings from the student survey. First, characteristics of survey respondents were 
compared with the characteristics of the full study sample. Doing so provides an indication of 
how representative the survey respondents are of the full study sample, one way to consider 
whether the survey results generalize to the full study sample in light of survey nonresponse. 
Second, characteristics of program group survey respondents were compared with characteristics 
of control group survey respondents. Doing so provides an indication of how similar program and 
control group survey respondents were to each other at the outset of the study, one way to assess 
whether differences between the groups’ later outcomes can be interpreted as being the result of 
CUNY Start. 


1A small number of program and control group members were not surveyed because their informed consent 
forms were not yet at MDRC when the survey sample was selected. 
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It is important to note that many survey questions were asked of subsets of survey re- 
spondents only (for example, those who took math classes during the semester after random as- 
signment). In those cases, it is generally not warranted to interpret differences in later outcomes 
between the subsets of program and control group respondents as being caused by CUNY Start. 


Comparison of Respondent and Full Sample Baseline Characteristics 


Survey respondents and the full evaluation sample have many statistically significant dif- 
ferences with respect to their baseline characteristics. However, these statistically significant dif- 
ferences do not indicate any practically significant differences with respect to the representative- 
ness of survey respondents. For example, compared with the full sample, survey respondents are 
1.0 percentage points less likely to have a parent with a college degree and 0.9 percentage points 
more likely to have a parent who is not a high school graduate — differences that are statistically 
significant but practically not very meaningful. 


The full sample and survey respondents exhibit the greatest dissimilarity with respect to 
gender; however, the difference is again not practically very meaningful. Women represent 50 
percent of survey respondents and 47.5 percent of the full sample. Thus, women are overrepre- 
sented among survey respondents by 2.5 percentage points compared with the full sample. Com- 
paring the characteristics of student survey respondents with the full evaluation sample provides 
some evidence that the survey results may generalize to the full sample. 


Comparison of Program and Control Group Respondent Baseline 
Characteristics 


Nearly identical proportions of the program and control groups responded to the survey 
(75 percent). With respect to their baseline characteristics, respondents in the two groups are 
highly similar. Survey respondents in the program and control groups only exhibited statistically 
distinguishable differences with respect to whether they had earned an occupational or technical 
certificate before random assignment. It is reasonable to compare the program and control groups 
and expect an internally valid estimate of CUNY Start’s effect on survey respondents. 


Creation of Student Survey Scales 


This section of the appendix discusses two scales from the student survey reported in 
Chapter 2. Specifically, this section provides details on the “quality-of-seminar” scale and the 
“quality-of-advising” scale, including the questions that made up the scales and the data pro- 
cessing conducted to calculate the values presented in Chapter 2. 
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Quality-of-Seminar Scale 


The quality-of-seminar measure presented in Table 2.8 is derived from five questions 
administered in the student survey. Students were asked to indicate whether they (1) strongly 
disagreed, (2) disagreed, (3) neither disagreed nor agreed, (4) agreed, or (5) strongly agreed with 
the following statements: 


Your seminar provided you with practical experience for navigating the university 
and financial aid systems. 


Your seminar supported your learning in your other course(s). 


Your seminar provided important opportunities for you to present on topics of 
interest. 


Your seminar improved your confidence in your ability to achieve your academic 
goals. 


Your seminar helped improve your study or time management skills. 


A scale was created as the unweighted average of a student’s responses to all five ques- 
tions. Respondents had the option to skip or refuse any question in the scale; some students re- 
sponded to some, but not all the questions in the scale. The scale was only calculated for students 
who answered all five items. The table presents the percentage of students who averaged a score 
of 4 or higher. 


Finally, a factor analysis tested how well the items included in the scale measure a com- 
mon underlying construct. The factor analysis yielded a standardized Cronbach’s alpha of 0.91, 
meaning that they did measure a common underlying construct. 


Quality-of-Advising Scale 


The quality-of-advising measure presented in Table 2.8 is derived from 11 questions ad- 
ministered in the student survey. The first five questions were adapted from the Academic Ad- 
vising Inventory.” Students were asked to indicate whether they (1) strongly disagreed, (2) disa- 
greed, (3) neither disagreed nor agreed, (4) agreed, or (5) strongly agreed with the following 
statements: 


You are satisfied in general with the academic advising you have received. 


You have received accurate information about courses, programs, and require- 
ments through academic advising. 


Winston and Sandor (n.d.). 
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Academic advisors kept you informed about deadlines related to institutional pol- 
icies and procedures, such as drop/add periods, withdrawal deadlines, registration 
periods, etc. 


Academic advising has been available when you needed it. 
Sufficient time has been available when you met with academic advisors. 


You often discussed your academic goals and progress toward those goals with 
your advisor. 


Your academic advisor acknowledged your accomplishments. 
Advising has given you the tools needed to make sound academic choices. 


Advising has enabled you to seek the resources required to achieve your academic 
goals. 


Advising has given you the confidence to advocate for yourself as you navigate 
the university system. 


You feel more prepared to pursue your academic goals and navigate the university 
system as a result of the advising you have received. 


A scale was created as the unweighted average of a student’s responses to all 11 ques- 
tions. Respondents had the option to skip or refuse any question in the scale; some students re- 
sponded to some, but not all the questions in the scale. The scale was only calculated for students 
who answered all 11 items. The table presents the percentage of students who averaged a score 
of 4 or higher. 


Finally, a factor analysis tested how well the items included in the scale measure a com- 
mon underlying construct. The factor analysis yielded a standardized Cronbach’s alpha of 0.958, 
meaning that they did measure a common underlying construct. 


Instructor Survey 


Survey Fielding and Respondent Sample 


The instructor survey asked CUNY Start math and reading/writing instructors as well as 
developmental math, reading, and writing instructors about a variety of topics including their 
backgrounds, instructional delivery, strategies, professional development, and use of time. CUNY 
Start instructors were also asked about their CUNY Start apprenticeships. Initially, instructors 
were sent multiple email messages encouraging them to complete the survey online. Later, in- 
structors who did not complete the survey online were called on the phone to complete it in a 
computer-assisted telephone interview. Abt Associates led the fielding of the survey. 
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The survey was fielded to all CUNY Start math and reading/writing instructors 
(Ncs-—surveyea = 66) and a random sample of non-CUNY Start developmental math, reading, 
and writing instructors (Nnon—cs—surveyea = 276). Fielding occurred during spring 2015, fall 
2015, and spring 2016, starting in April or October (depending on the cohort) and lasted approx- 
imately 6 to 10 weeks. Once an instructor responded to the survey, that instructor was not sur- 
veyed again in future semesters. Instructors who were selected for the survey in a semester and 
did not respond to it in that semester were fielded the survey again in the next semester that they 
taught. A total of 216 responses were collected (57 CUNY Start instructors and 159 non-CUNY 
Start instructors), an overall survey response rate of 63.2 percent; 86.4 percent of CUNY Start 
instructors responded and 57.6 percent of non-CUNY Start instructors responded. Due to the rel- 
atively low response rate of non-CUNY Start instructors, the non-CUNY Start instructor survey 
results must be interpreted cautiously — it is quite possible that non-CUNY Start survey respond- 
ents are not representative of all non-CUNY Start instructors. 


Information on nonrespondent instructors is limited to their colleges, the terms they 
taught during the study period, and the subjects they taught. Because this information is so lim- 
ited, this report does not comment on the representativeness of the instructor survey. 
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Appendix E 


Impact-Estimation Model Specifications and 
Additional Analyses 


The Effect of the Option to Participate in CUNY Start 


The main analyses presented in Chapter 3 are intent-to-treat (ITT) estimates. To conduct these 
analyses, a generalized least squares estimator is used to estimate the effect of the opportunity to 
particpate in CUNY Start.' The following model specification is used: 


Y¥;= an a;Block;; + ByP; + Dar VieXni + €i (1) 


Here, Y; represents a target outcome (such as being deemed college-ready in math or not). Block;; 
is a vector of J random assignment block indicators equal to 1 if student i is in block j and 0 
otherwise.” P; is a binary program assignment indicator equal to 1 if student i is randomly as- 
signed to the program group and 0 otherwise. X;; is a vector of K/2 student baseline character- 
istics and K /2 missing dummy indicators, one for each baseline variable. Baseline characteristics 
include pretest scores and random assignment date.* The ¢; are the residual variances. Robust 
standard errors are used. 


The parameter of interest is £,, the average effect of program assignment on Y;. Since 
program assignment is random and weights are used to account for the changing random assign- 
ment ratio (see below), the estimator of £, is unbiased with respect to the average effect of the 
ITT.4 


Subgroup Analyses 


To test for differences in subgroup impacts, split-sample regression analyses were conducted and 
the Hr statistic used to determine statistical significance.° This approach involves estimating the 
ITT effect for each group separately (for example, full-timers and part-timers) and then determin- 
ing whether the amount of variation in estimated effects among the groups is greater than what 
would be expected to be seen by chance alone. 


‘For the main analyses, SAS’s SURVEYREG procedure is used. For the subgroup analyses, SAS’s GLM 
procedure is used. 


?Fach block indicator represents a college x cohort x intended program type. The study includes four col- 
leges and three cohorts. Three of the four colleges include two program types (full time and part time) and the 
fourth college only includes a full-time program. Thus 21 unique blocks are included. 

3Random assignment date is included as the number of days before the close of registration that the student 
was randomly assigned. 

“The estimand is the ITT effect for the average person, not the effect for the average site or average random 
assignment block. 

°Greenberg, Meyer, and Wiseman (1994). 
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Weights for the Analyses (Chapter 3) 


Weights are used in the main analyses to account for the changing random assignment ratio.° 
Weights are calculated to make the effective (weighted) random assignment ratio the same in all 
random assignment blocks. The effective random assignment ratio is equal to the full sample’s 
random assignment ratio. Weights are calculated as follows: 


w=) v0-n0( 2) 


e Pi; is equal to 1 if individual i in random assignment block j is assigned to the 
piogiaia group and 0 if assigned to the control group. 


Where: 


e P.; is equal the proportion of sample members in random assignment block j 
assigned to the program group (that is, the average value of P;; in random as- 
signment block /). 


e P. is equal the proportion of ALL sample members randomly assigned to the 
program group (that is, the average value of P;; across all sample members). 


Weights for the Student Survey Analyses (Chapter 2) 


Survey weights are used in the survey analyses to account for the fact that the survey was fielded 
to nearly all control group members, but to a random sample of program group members. Survey 
weights are calculated to make the effective (weighted) random assignment ratio the same in all 
random assignment blocks and to make each block be weighted proportionately to its sample size 
in the full evaluation sample. Weights are calculated as follows: 


P. P. 
Wee = >, (sersc7) + (1- Pi;) = a) * 
j PB; 


e Pi; is equal to 1 if individual i in random assignment block j is assigned to the 
pineal group and 0 if assigned to the control group. 


uy 


= 
survey 


nsurvey 


Where: 


°The random assignment ratio was different, by design, for each cohort. The changes were made due to 
different expectations in the fall and spring with respect to how many students could be recruited, and with 
respect to the number of program slots. 
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pe“"’eY is equal to the proportion of sample members in random assignment 


block j who were fielded a survey and assigned to the program group. 


e P. is equal the proportion of all sample members randomly assigned to the 
program group. 


e 7, is the size of the analytic sample in random assignment block j. 


e nis the total size of the analytic sample. 


surve 
n; ” 
ee) 

j. 


is the size of the sample fielded a survey in random assignment block 


e ns“rvey is the total size of the analytic sample fielded a survey. 


The survey weights do not attempt to account for survey nonresponse. 


Method for the Treatment-on-the-Treated (TOT) Analyses 


Two-stage least squares is used to conduct the TOT analyses. The logic of the TOT analysis is as 
follows: The difference in the average outcomes of all program and control group members is an 
unbiased estimate of the average ITT. However, because only 81 percent of eligible students par- 
ticipated in the program, this difference is not the effect of the TOT. Instead, the ITT is a weighted 
average of an assumed zero average effect for the 19 percent who were “no-shows” and the av- 
erage effect of the TOT for the 81 percent who participated in CUNY Start. Through simple 
algebra and the assumption of zero treatment effect on the “no-shows,” it is possible to estimate 
the effect of the TOT. The following formula can be used: 


1.00 * ITT_Effect = 0.81 * TOT_Effect + 0.19 * 0 (1) 


Solving for TOT_Effect yields that TOT_Effect = ITT_Effect / 0.81.’ The effect of the treatment 
on the treated is simply the ITT effect divided by the participation rate. Two-stage least squares 
through the procedure described by Brachet is used to compute robust standard errors for the TOT 
effect estimate.® 


7Bloom (1984). 
®8Brachet (2007). 
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Appendix F 


What Works Clearinghouse-Style Impact Table 


This table displays information similar to that contained in Table 3.2. The differences are that this 
table only includes continuous outcome variables and that it provides the standard deviations of 
the outcomes, which allows one to translate estimated effects in their original units into effect- 
size units. This information may be of interest to meta-analysts and to the What Works Clearing- 
house. 


Appendix Table F.1 


Continuous Educational Outcomes in 
What Works Clearinghouse Format 


Program Group Control Group Estimated Effect 
Standard Standard Standard 

Outcome Mean Deviation Mean Deviation Difference Error P-Value 
End of the program semester 
Number of college-ready subjects 
(out of 3) 1.874 1.091 1.379 0.990 0.49 0.032 <0.0001 
College-level credits attempted 0.81 2.71 3.56 4.21 -2.75 0.13 <0.0001 
College-level credits earned 0.56 2.16 2.45 3.65 -1.88 0.12 <0.0001 
Postprogram semester 
College-level credits attempted 6.98 6.44 5.20 5.52 1.78 0.21 <0.0001 
Sample size (total = 3,835) 2,997 838 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY's Institutional Research Database, and test data from CUNY's Administrative Data 
Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

All values are weighted to account for random assignment ratios that vary across random 
assignment blocks. 

See Appendix E for details on the impact-estimation model. 
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Appendix G 


Exploring the Effects of Participating in CUNY Start 


The primary focus of Chapter 3 is answering the question, “What is the average effect of the 
option to participate in CUNY Start?” Analyses that answer this question are known as “intention- 
to-treat” (ITT) analyses. ITT analyses compare the average outcomes of all individuals randomly 
assigned to the program group with the average outcomes of all individuals randomly assigned 
to the control group, regardless of participation in CUNY Start (for program group members) or 
enrollment in non-CUNY Start courses (for control group members). The ITT effect is the aver- 
age effect of the opportunity to participate in CUNY Start. The ITT effect reflects the reality that 
not all individuals assigned to a program comply by participating in it. In some cases, the ITT 
effect may reflect the effect that a policymaker can realistically expect to achieve, given the likely 
scenario that not everyone offered a program will participate in it. 


In this appendix, two additional questions are explored: 


1. What is the average effect of participating in CUNY Start, compared with what oth- 
erwise would have happened?! 


Analyses that answer this question are known as “treatment-on-the-treated” (TOT) anal- 
yses. The TOT analyses estimate the effect of participating in CUNY Start (as a nonmatriculated 
student) compared with whatever educational experiences otherwise would have occurred, 
whether those be immediately enrolling in non-CUNY Start courses (as a matriculated student) 
or not enrolling in college at all. The second question addressed is: 


2. What is the average effect of participating in CUNY Start (as a nonmatriculated 
student), compared with enrolling in non-CUNY Start courses (as a matriculated 
student)?? 


These second analyses estimate the effect of participating in CUNY Start compared with 
a specific alternative — immediately enrolling in non-CUNY Start courses. This analysis is par- 
ticularly interesting because it is important to understand how much of CUNY Start’s effect on 
college readiness and college-level credit accumulation is due to its core classroom components 
and support services (as opposed to getting students to enroll). 


More details on the importance of each of these questions are provided below. The dis- 
tinction between these questions is elaborated upon at the start of the second set of analyses. These 
analyses are more technical than those presented in the main chapters of this report — readers 


'This effect is estimated among the subset of students who participate in CUNY Start if they are assigned to 
the program group, regardless of what they would do if they were assigned to the control group. 

°This effect is estimated among the subset of students who participate in CUNY Start (as nonmatriculated 
students) if they are assigned to the program group, and who immediately enroll in non-CUNY Start courses (as 
matriculated students) if they are assigned to the control group. 
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primarily interested in the substantive implications should skip to the summary at the end of this 
appendix. 


The Effect of Participating in CUNY Start: TOT Analyses 


This section addresses the following question: What is the effect of participating in CUNY Start 
among program group students who chose to participate? This question may be of great interest 
to those who operate CUNY Start and would like to understand its effect on those who experience 
the program. 


Recall that 19 percent of individuals who were randomly assigned to the program group 
did not participate in CUNY Start (see Table 3.1). In the experimental literature these individuals 
are often referred to as “no-shows.” Below, for a limited set of outcomes, estimates are provided 
of the average effect of participating in CUNY Start (TOT) among the 81 percent of program 
group members who did participate in CUNY Start. These analyses rely on stronger assumptions 
(described in the analyses section below) than the main ITT analyses presented in Chapter 3; 
therefore, caution is required when interpreting these results, and they are considered 
exploratory.? 


The main TOT findings are: 


e Some students who chose to participate in CUNY Start would not have en- 
rolled at City University of New York (CUNY) colleges had they not been 
offered the option of CUNY Start. 


e CUNY Start participation had a large, positive effect on the number of subjects 
in which students were college-ready at the end of the program semester. 


e In alignment with CUNY Start’s theory of change, students participating in 
CUNY Start earned fewer college-level credits during the program semester 
than they otherwise would have. 


3On a related note: Recall that the opportunity to participate in CUNY Start (the ITT effect, presented in 
Chapter 3) causes more students to enroll at a CUNY college (that is, to participate in CUNY Start or to enroll 
in any non-CUNY Start courses as matriculated students). Similarly, participating in CUNY Start (TOT) causes 
more students to enroll at a CUNY college. Had they not participated in CUNY Start, some CUNY Start partic- 
ipants would have elected not to enroll at a CUNY college. Such students are still included in these analyses. 
Consequently, the TOT analyses that examine outcomes such as college readiness and credits earned are influ- 
enced by this initial effect on getting students to enroll at a CUNY college. 
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These results mirror the ITT findings, though the effects found are larger and less certain. 
More information on the methods used for the TOT analyses is provided in Appendix E. 


Before describing the results in more detail, it is important to highlight a critical assump- 
tion in these analyses: the TOT analyses rely on the assumption that the outcomes of the program 
group “no-shows” (the 19 percent of the program group who did not participate in CUNY Start) 
are the same as they would have been had these students been randomly assigned to the control 
group.‘ In this study, the no-shows comprise two groups: the 8 percent of the program group who 
enrolled in non-CUNY Start courses at CUNY colleges (and did not participate in CUNY Start) 
and the 11 percent of the program group who did not enroll in a CUNY college (that is, they did 
not participate in CUNY Start or enroll in any other courses at a CUNY college). 


Is the assumption reasonable? It is difficult to say for sure. Individuals in the evaluation 
who are randomly assigned to the program group have three options: (1) participate in CUNY 
Start, (2) enroll ata CUNY college in non-CUNY Start courses (as matriculated students), or (3) 
do not enroll at a CUNY college. Individuals who are randomly assigned to the control group 
only have options 2 and 3. Program group members who elect not to participate in CUNY Start 
(option 1) are left with the same two options as they would have had if assigned to the control 
group; therefore, rationally, they might be expected to behave the same way they would have had 
they been assigned to the control group in the first place (and thus end up with the same 
outcomes). 


However, practically, there are plausible reasons this assumption may have been violated. 
For example, a student who is randomly assigned to the program group may initially sign up for 
CUNY Start and soon after realize he or she does not want to participate because of the program’s 
intensity, because of discomfort with its pedagogy, etc. By the time a student makes this decision, 
the option to enroll in non-CUNY Start courses may have changed — desired courses may have 
filled up, financial aid deadlines passed, etc. Consequently, such a “no-show” student may not 
end up enrolling in non-CUNY Start courses in that first semester even though, had the student 
been assigned to the control group initially, he or she would have enrolled in non-CUNY Start 
courses. Thus, the outcomes of the no-shows would have been different had they originally been 
assigned to the control group, violating the assumption. 


Due to this fairly strong assumption, the results of the TOT analyses are considered 
exploratory. 


Appendix Table G.1 presents the estimated effects of participating in CUNY Start on a 
few selected academic outcomes, among those program group members who participated in 


‘This assumption is known as “the exclusion restriction” in the economics literature. 
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Appendix Table G.1 


Effects of Participating in CUNY Start (Treatment-on-the-Treated) 


CUNY Control 
Start Group 
Outcome Participants Counterparts Difference P-Value 
Enrollment during the program semester 
Enrolled at a CUNY college (%) 100.0 87.2 12.8 *** <0.001 


Educational achievement by the end of the program semester 


Number of college-ready subjects (out of 3) 2.03 1.41 0.62 *** <0.001 
College-level credits earned 0.1 2.5 -24 ** <0.001 
Sample size (total = 3,835) 2,997 838 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, data 
from CUNY's Institutional Research Database, CUNY Start program participation data, and test data from 
CUNY's Administrative Data Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random assignment 
blocks. 


Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Sample size refers to the number of students included in the analyses, not the number of CUNY Start 
enrollees in the program group. 


CUNY Start, referred to as “CUNY Start participants.” By construction, the effect estimates pre- 
sented in Appendix Table G.1 are 24 percent higher than the ITT effects on the same outcomes 
presented in earlier tables (ITT / 0.81). 


Because TOT effects are estimated among CUNY Start participants, 100 percent of 
CUNY Start participants enrolled at a CUNY college. Among CUNY Start participants, an esti- 
mated 12.8 percentage points would not have enrolled at CUNY colleges had they been assigned 
to the control group (thus the control group counterparts are estimated to have enrolled at a rate 
of 87.2 percent). Whereas the ITT effect estimate on enrolling (that is, participating in CUNY 
Start or enrolling in any non-CUNY Start courses) was 10.3 percentage points (see Table 3.1), 
the TOT effect estimate is 12.8 percentage points, 24 percent higher than the ITT effect. 


With respect to students’ college readiness, by the end of the program semester CUNY 
Start participants were college-ready in 2.03 subject areas (out of 3) on average, whereas their 
control group counterparts were college-ready in an estimated 1.41 subject areas, for an estimated 
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TOT effect of 0.62 subject areas.° Finally, CUNY Start’s estimated TOT effect on earning col- 
lege-level credits is -2.4 credits. As is to be expected, the general substantive story presented in 
the TOT analyses is similar to that resulting from the ITT analyses, only more dramatic. Through 
the end of one semester, CUNY Start has a large, positive effect on college readiness, and a large, 
negative effect on earning college-level credits. 


The Effect of Participating in CUNY Start Compared with 
Enrolling in Any Non-CUNY Start Courses 


This appendix next examines the estimated effect of participating in CUNY Start compared with 
one specific alternative: immediately enrolling in non-CUNY Start courses at a CUNY college 
as a matriculated student. The effect is estimated through one semester after random assignment, 
among the subset of students who immediately enroll in non-CUNY Start courses if they are 
assigned to the control group, and who participate in CUNY Start if they are assigned to the 
program group.° 


At first glance, the effect of participating in CUNY Start compared with immediately 
enrolling in non-CUNY Start courses may sound the same as the effect of participating in CUNY 
Start, presented in the previous section. However, there is at least one critical difference. The 
effect of participating in CUNY Start (TOT) presented previously is defined as the difference 
between two potential outcomes: 


1. The outcomes of all program group members who participated in CUNY Start 


2. The outcomes of program group members who participated in CUNY Start, had they 
been assigned to the control group 


Evidence shows that some program group members who participated in CUNY Start 
would not have immediately enrolled in non-CUNY Start courses had they been assigned to the 
control group — that is, they would not have enrolled at any CUNY college. These students are 
included in the TOT analyses and are part of the TOT effects presented in in the previous section. 
In contrast, the analyses presented next seek to estimate the effect of participating in CUNY Start 


°For simplicity, Appendix Table G.1 presents the TOT effects for a limited set of outcomes only. To estimate 
the TOT effects for the other college-readiness measures presented in the second panel of Table 3.2, divide the 
ITT effects by 0.806. 

SPart of what is described in this appendix is a theoretical framework (called principal stratification) that is 
helpful for understanding the assumptions and analyses conducted to estimate the effect of interest. While the 
framework involves theory, the goal of understanding the effect of participating in CUNY Start compared with 
immediately enrolling in non-CUNY Start courses is of practical significance. 
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compared with enrolling in non-CUNY Start courses (excluding anyone who would not enroll at 
all). 


Part of the ITT and TOT effects on college readiness and credit accumulation stem from 
CUNY Start’s initial effect of getting more students to enroll at CUNY colleges. While that effect 
is important, getting more students to enroll in college is probably not a consequence of some of 
the most interesting components of CUNY Start — those that occur in the classroom (that is, 
CUNY Start’s intensity, pedagogy, and curriculum) and the student support services (for exam- 
ple, advising). 


Thus, another question of great interest is, “What is the effect of participating in CUNY 
Start compared with immediately enrolling in non-CUNY Start courses at a CUNY college?”’ 
This question is an attempt to separate the portion of CUNY Start’s effects that stem from its 
effect on enrollment from the portion that can be attributed to the classroom components and 
support services. The question examined here is thus, “What is the effect of the classroom com- 
ponents and support services of CUNY Start?” 


Random assignment ensures that the full program group and full control group are not 
systematically different at the outset of the study. The challenge with answering the question 
presented in the previous paragraph is that only 81 percent of students assigned to the program 
group participated in CUNY Start and 78 percent of students assigned to the control group en- 
rolled in non-CUNY Start courses. Random assignment does not ensure that these two subsets of 
students were similar at the start of the study, so the differences between the two groups’ later 
outcomes could be a result of CUNY Start experiences compared with the experiences students 
would have had in non-CUNY Start courses, or they could reflect differences between the types 
of students who ended up participating in CUNY Start and those who ended up enrolling in non- 
CUNY Start courses. Below, two approaches are used to attempt to answer the question above. 


Approach 1: Compare Program Group CUNY Start Participants with 
Control Group Students Enrolled in Non-CUNY Start Courses 


One approach to estimating the effect of participating in CUNY Start compared with 
enrolling in non-CUNY Start courses (as a matriculated student) is to take the difference between 
(1) the average outcomes of the 81 percent of program group members who participated in CUNY 
Start (CUNY Start participants) and (2) the average outcomes of the 78 percent of control group 
members who enrolled in any non-CUNY Start courses as matriculated students (referred to here- 
after as “college matriculates”). As noted above, there is a risk that this difference may be biased 


7Note that, among the questions addressed within this report, the question that these analyses seek to answer 
is most closely aligned with the questions addressed in CUNY’s internal propensity-score-matching analysis of 
CUNY Start. See Allen and Horenstein (2013). 
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(with respect to the desired inference) because these two groups may have been systematically 
different before the study began. 


One assurance that this potential bias is probably small comes from the U.S. Department 
of Education’s What Works Clearinghouse (WWC) guidance on attrition in individual-level ran- 
domized controlled trials such as this study.® The analysis described here can be thought of as an 
overall attrition rate of 20 percent, with differential attrition of 3 percentage points. According to 
the WWC criteria, this level of attrition would be considered low, with a low potential for bias 
when comparing these two groups. 


A second assurance that the potential bias may be small comes from comparing the meas- 
ured baseline characteristic of the two groups. Such a comparison finds that most of the two 
groups’ measured baseline characteristics were similar,° with the following exceptions: 


e Compared with CUNY Start participants, college matriculates in the control 
group were 6 percentage points more likely to say English was their primary 
language. 


e Compared with CUNY Start participants, college matriculates in the control 
group were 5 percentage points more likely to have applied for financial aid. 


e Compared with CUNY Start participants, college matriculates in the control 
group were 4 percentage points less likely to be triple remedial. 


e Compared with CUNY Start participants, college matriculates in the control 
group were 5 percentage points more likely to be college-ready in writing. 


These general similarities suggest low bias, although it is still theoretically possible that 
important differences exist between the two groups with respect to unmeasured characteristics. 


Appendix Table G.2 presents the estimated effects at the end of the program semester of 
participating in CUNY Start compared with enrolling in non-CUNY Start courses, on two out- 
comes: college readiness and college-level credits earned. The analyses compare program group 
CUNY Start participants with college matriculates in the control group. 


Analyses that compare the regression-adjusted outcomes for these two groups show that, 
after one semester, CUNY Start participants were college-ready in 0.54 more subject areas (out 


8U.S. Department of Education, Institute of Education Sciences (2014). 


In addition to the four bulleted characteristics, all characteristics presented in Appendix Table A.1 were 
examined. No other characteristics yielded statistically significant differences between CUNY Start participants 
and college matriculates in the control group. 
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Appendix Table G.2 


Educational Achievement for CUNY Start Participants Compared with 
Control Group College Matriculates 


CUNY Control Group 


Start College 
Outcome Participants Matriculates Difference P-Value 
Enrollment during the program semester 
Enrolled at a CUNY college (%) 100.0 100.0 0.0 - 


Educational achievement by the end of the program semester 


Number of college-ready subjects (out of 3) 2.05 1.51 0.54 *** <0.001 
College-level credits earned 0.2 3.0 -2.9 ** <0.001 
Sample size (total = 3,077) 2,416 661 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, data 
from CUNY's Institutional Research Database, CUNY Start program participation data, and test data from 
CUNY's Administrative Data Warehouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random assignment 
blocks. 


Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


of 3) than college matriculates in the control group (2.05 compared with 1.51 college-ready sub- 
ject areas). This effect estimate is similar in magnitude to the ITT effect estimate on the same 
outcome (see Table 3.2). It suggests that the ITT effects on college readiness are not simply a 
result of getting more students to enroll in college, but rather that compared with enrolling in non- 
CUNY Start courses, participating in CUNY Start has large effects on college readiness. 


Simultaneously, after one semester CUNY Start participants earned 2.9 fewer college- 
level credits (around one full course’s worth) than college matriculates in the control group (0.2 
compared with 3.0 college-level credits). This -2.9 credit effect is a full credit larger than the ITT 
effect estimate of -1.9 credits. There are two reasons for this shift. First, college matriculates in 
the control group earned more college-level credits than control group members who did not en- 
roll in any college. Second, program group CUNY Start participants earned fewer college-level 
credits than did program group members who did not participate in CUNY Start, many of whom 
enrolled in non-CUNY Start courses. As a result, the negative effect comparing the full program 
group with the full control group is smaller than the negative effect comparing program group 
CUNY Start participants with college matriculates in the control group. 
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In sum, this analysis provides suggestive evidence that after one semester, participating 
in CUNY Start, compared with enrolling in non-CUNY Start courses as a matriculated student, 
has a large positive effect on college readiness and a large negative effect on college-level credit 
accumulation — reinforcing the notion that this program offers a short-term trade-off whose ul- 
timate success will not be known without longer-term follow-up data collection. 


Approach 2: Principal Stratification and Bounding 


A second approach to addressing the question, “What is the effect of participating in 
CUNY Start compared with enrolling in any non-CUNY Start courses ata CUNY college?” relies 
on a theoretical framework known as principal stratification. '° This framework helps clarify the 
assumptions that go into answering this question, assumptions that are more opaque using the 
first approach. As will be shown, the general results are similar to those found using Approach 1. 


Principal stratification describes individuals as belonging to one of many strata. Stratum 
membership is defined based on an individual’s potential values related to an intermediate be- 
havior under different states of the world. In the current example, stratum membership is defined 
based on the intermediate behavior “enrollment status,” which can take on three values: 


1. Donot enroll ina CUNY college [Don’t enroll (D)] 


2. Enroll in any non-CUNY Start courses at a CUNY college as a matriculated student 
[Matriculate (M)] 


3. Participate in CUNY Start [Participate in CUNY Start (C)] 


The “different states of the world” refer to being randomly assigned to the program group 
or control group. A person’s stratum membership is defined based on what that person’s enroll- 
ment status would be (a) when assigned to the program group, and (b) when assigned to the con- 
trol group. Stratum membership exists in principle, but cannot be observed in practice because it 
is not possible to simultaneously observe a person’s enrollment status when assigned to the pro- 
gram group and control group. Appendix Figure G.1 depicts the nine strata. 


Starting in the top left cell, students who belong to the Never Enroll (D-D) stratum would 
not enroll at a CUNY college (that is, they would not participate in CUNY Start and they would 
not enroll in non-CUNY Start courses) if assigned to the control group, and similarly would not 
enroll if assigned to the program group. As another example, students who belong to the D In- 
duced to C (D-C) stratum (bottom left) are those who would not enroll at a CUNY college if 


10Page et al. (2015); Frangakis and Rubin (2002). 
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Nine Strata Based on Enrollment Status 


Appendix Figure G.1 


No CUNY Start Offer 
(Control Group) 


4 ‘ Participate in CUNY 
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Start (C) (D-C) (M-C) (C-C) 


assigned to the control group, and would participate in CUNY Start if assigned to the program 
group. 


The research question addressed in this appendix is about one stratum — M Induced to 
C (M-C) — the effect of participating in CUNY Start compared with enrolling in non-CUNY 
Start courses, for the type of students who if assigned to the control condition would enroll in 
non-CUNY Start courses and if assigned to the program condition would participate in CUNY 
Start. 


Through a series of assumptions (described below) and the use of basic algebra, it is pos- 
sible to estimate the effect of participating in CUNY Start compared with enrolling in non-CUNY 
Start courses for the subgroup in that stratum, which is estimated to represent 71 percent of stu- 
dents in the evaluation. 


The assumptions are: 


e Defiers 1-2 do not exist.'' Defiers are people who participate in CUNY Start 
if they are assigned to the control group, yet do not participate in CUNY Start 
if they are assigned to the program group. The assumption that defiers do not 
exist is a “weak” assumption, meaning it seems very reasonable. 


Notably, within the control group, only 1 percent of students participated in CUNY Start. 
Thus, at most, defiers could represent 1 percent of the sample. What seems more likely, however, 
is that these students, who managed to participate in CUNY Start despite being assigned to the 
control group, would still have participated in CUNY Start had they been assigned to the program 


‘The term “defier” is borrowed from Angrist, Imbens, and Rubin (1996). 
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group. In other words, they are most likely Always Participate in CUNY Start (C-C), as is as- 
sumed here. 


e M Induced to D (M-D) do not exist. M Induced to D (M-D) are people who 
do not enroll at all if they are assigned to the program group (that is, they do 
not participate in CUNY Start and do not enroll in non-CUNY Start courses), 
but do enroll in non-CUNY Start courses if they are assigned to the control 
group. In other words, the assumption is that the option to participate in CUNY 
Start does not induce anyone who otherwise would have enrolled in non- 
CUNY Start courses to instead not enroll in at all. 


This is a “moderate” assumption, meaning it seems reasonable, yet there are plausible 
explanations for how it could be violated. It seems reasonable because it represents a seemingly 
inconsistent decision. If assigned to the control group, one can (1) enroll in non-CUNY Start 
courses at a CUNY college or (2) not enroll ata CUNY college. If assigned to the program group, 
one can choose between the control group’s two options or (3) participate in CUNY Start. For 
program group members who do not participate in CUNY Start, it is inconsistent for them to 
make a different decision among the first two options if assigned to the program group rather than 
the control group. 


In practice, however, it is plausible that some students in the program group signed up 
for CUNY Start, attended a class or two, did not like it and dropped CUNY Start, and then found 
it was too late to enroll innon-CUNY Start courses at a CUNY college, so ended up not enrolling 
ata CUNY college at all. Such a student, if assigned to the control group, might have immediately 
enrolled in non-CUNY Start courses as a matriculated student. Thus, there are plausible scenarios 
in which this assumption may have been violated. 


Notably, within the program group in this study, 11 percent of students did not enroll, so 
at most 11 percent of the sample could be M Induced to D (M-D). The assumption that M Induced 
to D do not exist, along with the weak no-defiers assumption, places the 11 percent in the Never 
Enroll (D-D) stratum. This assumption is probably the strongest one made in these analyses. 
However, since this assumption only affects 11 percent of all students, unless it is grossly violated, 
its influence on the results is likely to be fairly small. 


e D Induced to M (D-M) do not exist. D Induced to M (D-M) are people who 
enroll in non-CUNY Start courses if they are assigned to the program group, 
but who do not enroll at all if they are assigned to the control group. In other 
words, the assumption is that being offered CUNY Start does not induce any- 
one who otherwise would not have enrolled at all, to instead enroll in non- 
CUNY Start courses. 
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This is a weak to moderate assumption, meaning it seems very reasonable. D Induced to 
M (D-M) can enroll innon-CUNY Start courses as matriculated students whether they are offered 
CUNY Start or not, so it is hard to see why they would not enroll in those courses when they are 
assigned to do so (the control group), but would enroll in them when assigned to CUNY Start 
(the program group). However, there is at least one explanation of how this assumption could be 
violated: When a student is randomly assigned to the control group, the disappointment of not 
being offered CUNY Start could cause the student not to enroll at all. The same student, if instead 
offered CUNY Start, might feel good about the offer, sign up for CUNY Start, soon decide CUNY 
Start is too time-consuming, and then instead elect to enroll in non-CUNY Start courses. This 
hypothetical situation may have occurred, but it does not seem likely to have occurred very often. 


Notably, in this study’s program group only 8 percent of students enrolled in non-CUNY 
Start courses at CUNY colleges, so at most 8 percent of the sample could be D Induced to M (D- 
M). The assumption that D Induced to M do not exist, along with the no-defiers assumption, 
places the 8 percent of all students in the Always Matriculate (M-M) stratum. 


e CUNY Start has no effect on D-D, M-M, or C-C. It is assumed that those 
who Never Enroll (D-D), Always Matriculate (M-M), or Always Participate 
in CUNY Start (C-C) have the same outcome whether they are assigned to the 
program group or control group. This assumption seems plausible because un- 
der both scenarios a student’s enrollment status is the same. This assumption 
could have been violated for Always Matriculate (M-M), for example, if when 
assigned to the control group, a student immediately enrolls in non-CUNY 
Start courses. When assigned to the program group, that same student may 
only enroll in non-CUNY Start courses after trying out CUNY Start and de- 
ciding it is not for him or her. At that point non-CUNY Start courses may have 
filled up, so the student may perform differently than had he or she gone 
straight to the non-CUNY Start courses. 


e Outcome levels for individuals in D-C and M-C if assigned to the control 
group can be derived. The outcome levels of individuals in the control group 
who enroll innon-CUNY Start courses if they are assigned to the control group 
are known. Among these students, there are two strata — M-M and M-C (the 
second column in the figure) — and for the former group we also know the 
outcomes of individuals in M-M if they are assigned to the control group, 
based on the “no-effects” assumption in the previous bullet. Therefore, using 
algebra and the proportion of individuals in each of these two strata, the aver- 
age outcomes of students in M-C when assigned to the control group can be 
derived. A similar strategy can be used to derive the average outcomes of stu- 
dents in stratum D-C when assigned to the control group. 
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e Plausible outcome levels for individuals in D-C when assigned to the pro- 
gram group can be assumed. The outcome levels of individuals in strata D- 
C and M-C, when they are assigned to the program group (and thus participate 
in CUNY Start), cannot be uniquely determined for each stratum. However, 
the relationship between the outcomes of students in D-C and M-C when as- 
signed to the program group can be represented algebraically. This fact means 
that by making a range of reasonable assumptions about the outcome levels of 
individuals in stratum D Induced to C (D-C) when assigned to the program 
group, the outcomes of students in stratum M-C when assigned to the program 
group can be bounded. By comparing these bounds with the outcome levels of 
students in stratum M-C when assigned to the control group (see previous bul- 
let), one can also bound the impact of participating in CUNY Start for this 
stratum. 


These assumptions, and potential violations, leave room for some concern regarding the 
analyses and results that follow. However, the large magnitude of the effects presented and the 
relatively small proportions of students for whom each assumption is made (and thus may be 
violated) suggest that the pattern of findings presented is likely to be accurate, although the mag- 
nitude of the effect estimates may be further off than conventional uncertainty estimates suggest, 
due to the potential for bias. 


Results: 


First, it is estimated that approximately 71 percent of study enrollees fall into the stratum 
M Induced to C (M-C), comprising students who participate in CUNY Start if assigned to 
the program group and who enroll in non-CUNY Start courses at CUNY colleges if as- 
signed to the control group. This estimate is derived based on basic algebra, using the observed 
marginal proportions in Appendix Figure G.2. In Appendix Figure G.2, p,, is the proportion of 
students assigned to experimental condition z (1 = program group, 0 = control group), whose 
enrollment status is t (D = don’t enroll, M = matriculate, C = participate in CUNY Start). In the 
program group, 11 percent did not enroll (p;p = 0.11), 8 percent enrolled in non-CUNY Start 
courses as matriculated students (p,;, = 0.08), and 81 percent participated in CUNY Start 
(Pic = 0.81). In the control group, 21 percent did not enroll (ppp = 0.21), 79 percent enrolled 
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Appendix Figure G.2 
Marginal Percentages for the Strata 


No CUNY Start Offer 
(Control Group) 


Participate in CUNY 


Don’t pili ae we Start (C) 
Pop = VU. Pom — ¥. Doc = 0.01 
Don’t Enroll (D) Never Enroll M Induced to D Defiant 
oo 
£5 
O° : ; 
ro Matriculate (M) D Induced to M Always Matriculate atieR? 
# £ Pim = 0.08 (D-M) (M-M) 
6 
=D 
° ast : i : 
2 a Participate in CUNY DinducedioG M Induced to C Always Participate in 
Start (C) (D-C) (M-C) CUNY Start 
Pic = 0.81 (C-C) 


SOURCES: MDRC calculations using CUNY Start application data, MDRC random assignment data, 
data from CUNY’s Institutional Research Database, and CUNY Start program participation data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
All values are weighted to account for random assignment ratios that vary across random assign- 


ment blocks. 


in non-CUNY Start courses (Ygy = 0.79), and 1 percent participated in CUNY Start (Doc = 
0.01). The fact that 11 percent of the program group did not enroll, and the assumption of no M- 
Dandno Defier 1, together imply that 11 percent of the sample are Never Enroll (D-D). Similarly, 
the fact that 1 percent of program group members participated in CUNY Start, and the assumption 
of no Defier 1 or Defier 2, together imply that 1 percent of the sample are Always Participate in 
CUNY Start (C-C). Similar logic can be used to fill in the remaining strata, resulting in the esti- 
mate that 71 percent of the sample are M Induced to C (M-C). 


Second, the estimated effect of participating in CUNY Start compared with enrol- 
ling in non-CUNY Start courses on the number of developmental subject areas completed 
through one semester is between 0.57 and 0.78 subject areas (among M Induced to C (M- 
C)). On average, control group students who enrolled in non-CUNY Start courses were college- 
ready in 1.41 out of 3 courses after one semester, compared with an average of 1.98 to 2.19 among 
CUNY Start participants (among M Induced to C (M-C)). This large estimated effect is as large 
as (or larger than) the ITT effect on the same outcome. This estimated range is derived using 
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algebra similar to that described in the previous paragraph, along with assumed plausible bounds 
of the outcome levels for D Induced to C (D-C).” 


Third, the estimated effect of participating in CUNY Start compared with enrolling 
in non-CUNY Start courses on the college-level credits earned through one semester is be- 
tween -2.76 and -2.70 credits (among M Induced to C (M-C)). On average, control group stu- 
dents who enrolled in non-CUNY Start courses earned 2.85 college credits after one semester, 
compared with an average of 0.10 to 0.15 credits among CUNY Start participants (among M 
Induced to C (M-C)). This large estimated effect is larger than the ITT effect on the same outcome 
and similar to the TOT effect estimate. 


Summary 


In this appendix, two sets of analyses are presented: first, estimates of the effect of participating 
in CUNY Start compared with whatever CUNY Start participants would have done otherwise; 
second, estimates of the effect of participating in CUNY Start compared with enrolling in non- 
CUNY Start courses as a matriculated student. Both analyses require stronger assumptions than 
the basic ITT analyses that are the focus of Chapter 3. Both analyses draw very similar conclu- 
sions: After one semester, participating in CUNY Start has a large, positive effect on college 
readiness and a large, negative effect on college-level credit accumulation (as expected). The ITT 
analyses presented in Chapter 3, the TOT analyses presented at the start of the appendix, and the 
final analyses presented above draw similar conclusions. Nonetheless, there are some important 
distinctions. 


First, note that the ITT analyses presented in Chapter 3 and the TOT analyses presented 
at the start of this appendix suggest that one important benefit of CUNY Start is that the program 
gets more students to enroll in college (that is, to participate in CUNY Start or to enroll in non- 
CUNY Start courses). The program may have that effect because of its low cost, its unique ap- 
proach to curriculum and pedagogy, its student support services, or its promise to greatly reduce 


For the outcome number of developmental subjects completed, the lower-bound assumption is that D-C 
students, when assigned to the program group, are college-ready in 0.85 subjects at the end of the first semester. 
The lower the assumed outcome level for this group, the larger the estimated effect for M Induced to C (M-C). 
The chosen value was selected as a lower bound because: (a) this value is lower than the outcome level of any 
subgroup, and (b) this value is lower than the value in any stratum, except those who did not enroll. 

The upper-bound assumption is that D-C students, when assigned to the program group, are college-ready 
in 2.41 subjects at the end of the first semester. The higher the assumed outcome level for this group, the smaller 
the estimated effect for M Induced to C (M-C). The chosen value was selected as an upper bound because: (a) 
this value is well above the outcome level of any subgroup, and (b) this value is higher than the value in any 
other stratum. 
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students’ developmental course needs in a single prematriculation semester, or it may be an arti- 
fact of the experiment. 


Second, the TOT analyses provide important information on the outcome levels of 
CUNY Start participants and demonstrate even more dramatically than the ITT analyses the 
short-term trade-off of participating in CUNY Start. The outcome levels of CUNY Start partici- 
pants (rather than all program group members) may be of great interest to those who implement 
CUNY Start since programs typically track the outcomes of their participants, not all individuals 
who expressed interest in the program. The program group participant outcome levels are most 
likely to match the outcome levels CUNY Start staff members are accustomed to seeing, since 
they exclude the 19 percent of the program group who did not participate in CUNY Start. 


Finally, the estimated effects of participating in CUNY Start compared with enrolling in 
non-CUNY Start courses should provide reassurance that the ITT findings presented in Chapter 
3 (and the TOT findings presented earlier in this appendix) are not only a consequence of getting 
more students to enroll in college, but instead are largely a consequence of experiencing CUNY 
Start rather than non-CUNY Start courses. 


13Qne risk in the evaluation described in this report (and in many real-world evaluations) is that the effect 
on getting students to enroll in college could be an artifact of the evaluation itself, rather than a real benefit of the 
program. Ordinarily, eligible students who are interested in participating in CUNY Start are offered a spot in the 
program unless all spaces are full. During the study, however, some interested students were randomly assigned 
to the control group. The disappointment of not being offered CUNY Start could partly explain the positive 
effects on enrollment at a CUNY college. Attempts were made to ensure a smooth transition to matriculation at 
CUNY colleges for individuals randomly assigned to the control group; still, the hurdle of matriculating may 
have been higher than the hurdle to start participating in CUNY Start. 

A severe skeptic could be concemed that the effects on enrolling in college are the result of these factors. If 
that were true, then subsequent positive effects, which are in part a consequence of the effect on enrollment, 
could partly be explained by factors that are not at the core of CUNY Start (that is, its intensity, pedagogy, 
curriculum, and student support). For these reasons, the analyses presented in the second half of this appendix 
are an important supplement to the overall early results. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization 
dedicated to learning what works to improve the well-being of low-income people. Through 
its research and the active communication of its findings, MDRC seeks to enhance the 
effectiveness of social and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los 
Angeles, MDRC is best known for mounting rigorous, large-scale, real-world tests of new 
and existing policies and programs. Its projects are a mix of demonstrations (field tests 
of promising new program approaches) and evaluations of ongoing government and 
community initiatives. MDRC’s staff members bring an unusual combination of research 
and organizational experience to their work, providing expertise on the latest in qualitative 
and quantitative methods and on program design, development, implementation, and 
management. MDRC seeks to learn not just whether a program is effective but also how 
and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works 
across the social and education policy fields. MDRC’s findings, lessons, and best practices 
are shared with a broad audience in the policy and practitioner community as well as with 
the general public and the media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs 
for ex-prisoners, and programs to help low-income students succeed in college. MDRC’s 
projects are organized into five areas: 


* Promoting Family Well-Being and Children’s Development 
+ Improving Public Education 

* Raising Academic Achievement and Persistence in College 
¢ Supporting Low-Wage Workers and Communities 

« Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local 
governments, public school systems, community organizations, and numerous private 
philanthropies. 


